Version: ACE 4

ACE Components

The goal of this design document is to describe logical architecture of how ACE components interact with each other.

Logical

Please keep in mind that Sizing includes only resources needed by ACE components and do not include sizing for accommodating infrastructure (Operating System, Logging and Monitoring services, Cluster orchestration services, etc.).

PROD environment sizing is provided for 1000 concurrent users with 5 second think time. Where from ACE perspective the user is one API the call to the ACE Dynamic APIs. Please keep in mind that Sizing is provided for ACE overhead only. It can change Depending on complexity and availability of of depending systems, that ACE is accessing.

Sizing units are explained in this document. Total service requirements can be obtained by multiplying number of pod replicas with resources required.

In Sizing DEV/TEST and PROD means the environment type. If it is necessary to deploy both DEV and TEST environments, then required resources are 2 times the sizing provided below.

Services

note

All environment variable keys and values below are case-sensitive

Flow Manager

Flow Manager is the Single Page Application that itself contains only UI code for ACE Designer. It is deployed as the static "javascript/HTML" package.

Container name

digital-flow-ui

Deployment options

DEV/TEST	PROD
Change configuration	View configuration

Configuration options

PORT - port used by service.
LOG_LEVEL - Default info. service log verbosity level. Possible values error, warn, info, http, verbose, debug, silly.
REACT_APP_API_BASE_URL - URL of Orchestration service, where ACE Management APIs are available.
REACT_APP_PUBLIC_API_BASE_URL - URL of Orchestration service public endpoint, where ACE Dynamic APIs are available. If not set, then REACT_APP_API_BASE_URL is used.
REACT_APP_AUTH_STRATEGY - If this parameter is set Flow Manager will attempt to authenticate with Identity Provider, to obtain the JWT token, that will be used to access Management APIs. Currently supported authentication strategy is keycloak
REACT_APP_SHOW_SERVICE_CONFIG - Default false. Show service config tab in settings page. It allows to view service environmental variables for debugging purposes (requires that service has ENABLE_ENV_VAR_DEBUG set to true). Note that config feature is intended for use in non HA environment - if there are more than one service instance, then variables are returned from random instance.

Sizing

DEV/TEST (Pods, vCPU, RAM, Disk)	PROD (Pods, vCPU, RAM, Disk)
1, 200m, 64Mi, 5Gb	2, 200m, 128Mi, 5Gb

Scaling

This service can be scaled Horizontally, there is no persistent storage.

API Gateway

API gateway which is used to safely (by adding authentication and rate limiting policies for example) expose ACE services to the outside world.

It can also be used to add SSL certificate and expose ACE services via HTTPS protocol if it is not done by load balancer.

API gateway choice is client specific, most common options are cloud based solutions like Azure API Gateway or Amazon API Gateway. Good option for on-premise solution is Kong gateway.

Orchestration

Orchestration service exposes both ACE Management APIs and Dynamic APIs. It is essential in PROD to not expose this service to the outside world directly.

Management APIs are used by ACE Designer (Flow Manager) to change ACE configuration.

Dynamic APIs are APIs that are built by Domain Experts.

Orchestration service also deploys Scheduler Runtime.

Service has 2 endpoints:

Management endpoint - includes Management and Dynamic APIs. Management endpoint must not be exposed to outside world directly.
- Enabled - disabled by default. Can be enabled, see ACE_MANAGEMENT_API_DISABLED
- Port - PORT
- Timeout - ACE_MANAGEMENT_API_TIMEOUT, if not set, then fallback to ACE_API_TIMEOUT
Public endpoint - includes only Dynamic APIs. Intended to be exposed to outside world via API Gateway.
- Enabled - enabled by default. Can be disabled by un-setting PUBLIC_API_PORT
- Port - PUBLIC_API_PORT
- Timeout - ACE_API_TIMEOUT

Container name

digital-api-orch

Deployment options

DEV/TEST	PROD
Required	Required

Configuration options

PORT - port used by service for Management endpoint. 3000 by default.
PUBLIC_API_PORT - port used by service for Public endpoint. If port is not set, then Public endpoint is not started. 4000 by default.
ACE_MANAGEMENT_API_DISABLED - Default true. If set to false then Management endpoint is started.
LOG_LEVEL - Default info. Service log verbosity level. Possible values error, warn, info, http, verbose, debug, silly.
BASE_DAL_URL - URL of Data Access Layer service.
FLOW_ENRICHER_URL - URL of Flow Runner service.
FLOW_DESIGNER_URL - URL of Designer service.
DYNAMIC_API_URL - URL of Dynamic APIs service.
FLOW_VERSIONING_URL - URL of Versioning service.
CACHE_SERVICE - URL of Cache service.
DEPLOYED_VERSION - Version of the Container being deployed.
ACE_API_TIMEOUT - Default 0, no timeout. Request timeout in milliseconds for Public endpoint. Affects * *Management and/or Dynamic APIs** depending on service endpoint configuration.
ACE_MANAGEMENT_API_TIMEOUT - Request timeout in milliseconds for Management endpoint. Affects Management and/or Dynamic APIs, in seconds. If not set then ACE_API_TIMEOUT is used
JWT_SECRET - Private key used to sign Management API tokens.
JWT_EXPIRE - JWT token expiration period in minutes for Management API tokens.
ACE_UI_AUTH_STRATEGY - Management APIs security strategy. If set Management APIs will require a valid token to access them. Currently supported authentication strategy is keycloak.
KEYCLOAK_REALM - ACE ream in Keycloak.
KEYCLOAK_CLIENTID - ACE app client ID in Keycloak.
KEYCLOAK_CLIENT_SECRET - ACE app client Secret in Keycloak..
KEYCLOAK_AUTH_SERVER_URL - URL of Keycloak.
ACE_UI_URL - Public URL of Flow Manager, used for redirect from Keycloak.
SELF_EXTERNAL_URL - Public URL of orch service, used for redirect from Keycloak.
ENABLE_INSPECT - Default false. If true adds --inspect to entry-point, Debugger will listen on port 9239
ENABLE_ENV_VAR_DEBUG - Default false. If true, then enables to retrieve environmental variables for display in " Service config" tab.
ACE_LOG_EXCHANGE - (default: <empty>) - If set to true, logs request(url, method, body) and response(statusCode, body) related data at info level.

Sizing

DEV/TEST (Pods, vCPU, RAM, Disk)	PROD (Pods, vCPU, RAM, Disk)
1, 500m, 256Mi, 5Gb	2, 500m, 512Mi, 5Gb

Scaling

This service can be scaled Horizontally, make sure to deploy the same configuration to all instances when scaling.

Any scheduled tasks will execute as many times as how many Orchestration instances are deployed.

Designer

Designer service encapsulates application logic related to:

Designing of UI for the Flow Step
Managing of Workspace variables
Managing Scheduler jobs
Managing error handlers

Container name

flow-designer-server

Deployment options

DEV/TEST	PROD
Required	Required

Configuration options

PORT - port used by service.
LOG_LEVEL - Default info. Service log verbosity level. Possible values error, warn, info, http, verbose, debug, silly.
BASE_DAL_URL - URL of Data Access Layer service.
FLOW_VERSIONING_URL - URL of Versioning service.
ENABLE_INSPECT - Default false. If true adds --inspect to entry-point, Debugger will listen on port 9259
ENABLE_ENV_VAR_DEBUG - Default false. If true, then enables to retrieve environmental variables for display in " Service config" tab.

Sizing

DEV/TEST (Pods, vCPU, RAM, Disk)	PROD (Pods, vCPU, RAM, Disk)
1, 500m, 128Mi, 5Gb	2, 500m, 512Mi, 5Gb

Scaling

This service can be scaled Horizontally, make sure to deploy the same configuration to all instances when scaling.

Dynamic APIs

Dynamic APIs service implements application logic related to creating and managing of Dynamic APIs configuration (Operations, Schemas) as well as implement Dynamic APIs Runtime.

Dynamic APIs runtime:

Runs error handler logic
Maps Dynamic APIs to Flows

Container name

dynamic-api-server

Deployment options

DEV/TEST	PROD
Required	Required

Configuration options

PORT - port used by service.
LOG_LEVEL - Default info. Service log verbosity level. Possible values error, warn, info, http, verbose, debug, silly.
BASE_DAL_URL - URL of Data Access Layer service.
FLOW_ENRICHER_URL - URL of Flow Runner service.
ACE_API_TIMEOUT - Request timeout in milliseconds for Dynamic APIs, in seconds.
DYNAMIC_API_RESULT_PROP - (Removed in 4.22) node name in document returned by Flow that is returned as the output of the Flow, by default result.
ENABLE_ERROR_CONTEXT - Default true. If set to true then error context feature is enabled.
ENABLE_INSPECT - Default false. If true adds --inspect to entry-point, Debugger will listen on port 9249
CACHE_FLOWS - Default true. If true API configuration is cached to avoid retrieving it from Data Access Layer every time. If caching is enabled, then updates in Flow Manager are disabled unless CACHE_VALIDITY_REQUEST_ENABLED is set to true. It's not necessary to set this property explicitly, unless doing troubleshooting. If it is set to false, then ACE will have serious performance degradation! This is internal ACE cache, and is not related to data (response) caching in ACE.
CACHE_VALIDITY_REQUEST_ENABLED - Default true (since 4.39.0). Enables data updates from flow manager. This setting should be true in development environment and during deployment. As this feature has small performance hit, it can be set to false in production after deployment.
ENABLE_ENV_VAR_DEBUG - Default false. If true, then enables to retrieve environmental variables for display in " Service config" tab.
ACE_LOG_EXCHANGE - (default: <empty>) - If set to true, logs request(url, method, body) and response(statusCode, body) related data at info level.

Sizing

DEV/TEST (Pods, vCPU, RAM, Disk)	PROD (Pods, vCPU, RAM, Disk)
1, 500m, 256Mi, 5Gb	2, 500m, 512Mi, 5Gb

Scaling

This service can be scaled Horizontally, make sure to deploy the same configuration to all instances when scaling.

Flow Runner

Flow Runner implements Flow Runner Runtime. It is invoked either directly from Orchestration (when testing or debugging) or through Dynamic APIs (when calling Dynamic APIs).

Container name

flow-runner

Deployment options

DEV/TEST	PROD
Required	Required

Configuration options

PORT - port used by service.
LOG_LEVEL - Default info. Service log verbosity level. Possible values emerg, fatal, alert, crit, error, warning, warn, notice, info, debug, trace.
BASE_DAL_URL - URL of Data Access Layer service.
CACHE_SERVICE - URL of Cache service.
ACE_API_TIMEOUT - Request timeout in milliseconds for Flow Runner, in seconds.
CACHE_FLOWS - Default true. If true Flow configuration is cached to avoid retrieving it from Data Access Layer every time. If caching is enabled, then updates in Flow Manager are disabled unless CACHE_VALIDITY_REQUEST_ENABLED is set to true. It's not necessary to set this property explicitly, unless doing troubleshooting. If it is set to false, then ACE will have serious performance degradation! This is internal ACE cache, and is not related to data (response) caching in ACE.
CACHE_VALIDITY_REQUEST_ENABLED - Default true (since 4.39.0). Enables data updates from flow manager. This setting should be true in development environment and during deployment. As this feature has small performance hit, it can be set to false in production after deployment.
ENABLE_INSPECT - Default false. If true adds --inspect to entry-point, Debugger will listen on port 9229
RESPONSE_TIMEOUT - default value for rest step response timeout setting, see https://ladjs.github.io/superagent/#timeouts for the explanation
DEADLINE_TIMEOUT - default value for rest step deadline timeout setting, see https://ladjs.github.io/superagent/#timeouts for the explanation
ENABLE_EVAL_OPTIMIZATION - Default false. (Removed in 4.23) If set to true, then improves the performance by running optimized logic for object evaluation.
ENABLE_ENV_VAR_DEBUG - Default false. If true, then enables to retrieve environmental variables for display in " Service config" tab.
DISABLE_JS_CODE_STEP - Default false. If true, prevents execution of JS Code step (deprecated). Can be set as an additional safety measure if JS Code step is not required.
DISABLE_CODE_IVM_STEP - Default false. If true, prevents execution of JS Code step. Can be set as an additional safety measure if JS Code step is not required.
REST_STEP_MAX_FILE_SIZE_MB - Default 50. Specifies the max file size that REST Step will allow downloads for (in megabytes).
FULL_DATA_LOGGING - Default false. If enabled logs additional information like request/response bodies. Note that additional data may contain sensitive information like personally identifiable information or secrets.
ACE_FILE_REF_DOC_LIMIT_MB- (default 5) File size limit in MB for reading file content using file reference step.
ENABLED_EVAL_EXTENSIONS - (default "ivm,jsonata") Enabled expression evaluators. Possible values ivm, jsonata, eval. By default deprecated evaluator eval is disabled.
DEFAULT_EVAL_EXTENSION- (default "ivm") Default expression evaluator to be used when one is not specified in input.
IVM_TIMEOUT - (default "1000") Specific to ivm eval extension. Execution timeout in ms.
ACE_LOG_CORRELATION_ID (default: x-request-id) - Header name to retrieve correlation id from. Also used as header name for correlation header forwarding. Header value is NOT added in ACE 4 logs (that's done in ACE 5), but it is forwarded in network requests.
ACE_DISABLE_CORRELATION_HEADER_FORWARDING (default: <empty>) - if set to true, disables correlation header forwarding in network steps.
ACE_REQUEST_BODY_SIZE_LIMIT (default: 50mb*) - Sets limit for maximum request body size.

Sizing

DEV/TEST (Pods, vCPU, RAM, Disk)	PROD (Pods, vCPU, RAM, Disk)
1, 500m, 256Mi, 5Gb	2, 1, 1Gi, 5Gb

Scaling

This service can be scaled Horizontally, make sure to deploy the same configuration to all instances when scaling.

Cache

Cache service provides an abstraction on top of cache implementation layer. Currently supported caching implementation is Redis.

Container name

cache-service

Deployment options

DEV/TEST	PROD
Required if cache is used.	Required if cache is used.

Configuration options

PORT - port used by service.
LOG_LEVEL - Default info. Service log verbosity level. Possible values fatal, error, warn, info, debug, trace.
REDIS_CACHE_HOST - redis host.
REDIS_CACHE_PORT - redis port.
REDIS_CACHE_KEY - redis authentication key.
REDIS_KEY_PREFIX - prefix for redis cache keys (default: ace-cache).
REDIS_CACHE_DB - redis database number
REDIS_CACHE_USER - Redis username
REDIS_CACHE_TLS_ENABLED - Flag, set to true if redis is tls configured.
REDIS_CACHE_TLS_KEY - TLS private key
REDIS_CACHE_TLS_CERT - TLS cert
REDIS_CACHE_TLS_CA - TLS CA
REDIS_DEFAULT_TTL_MIN - Default 1440 (24h). Redis cache time to live before refreshing the result for an item (default TTL value if not specified for step).
ENABLE_INSPECT - Default false. If true adds --inspect to entry-point, Debugger will listen on port 9289
ENABLE_ENV_VAR_DEBUG - Default false. If true, then enables to retrieve environmental variables for display in " Service config" tab.

Sizing

DEV/TEST (Pods, vCPU, RAM, Disk)	PROD (Pods, vCPU, RAM, Disk)
1, 200m, 128Mi, 5Gb	2, 500m, 512Mi, 5Gb

Scaling

This service can be scaled Horizontally, make sure to deploy the same configuration to all instances when scaling.

Redis

Redis is the caching backend.

Container name

redis

Deployment options

DEV/TEST	PROD
Required if redis implementation for cache is used.	Required if redis implementation for cache is used.

Sizing

DEV/TEST (Pods, vCPU, RAM, Disk)	PROD (Pods, vCPU, RAM, Disk)
1, 50m, 128Mi, 5Gb	2, 500m, 512Mi, 5Gb

Scaling

Refer to the Redis documentation about how to scale the Redis.

Versioning

Versioning service encapsulates application logic relating to accessing Source Control Management systems or Exporting/Importing of the workspace configurations in general.

Container name

flow-versioning

Deployment options

DEV/TEST	PROD
Required	Required

Configuration options

PORT - port used by service.
LOG_LEVEL - Default info. Service log verbosity level. Possible values error, warn, info, http, verbose, debug, silly.
DYNAMIC_API_URL - URL of Data Access Layer service.
FLOW_ENRICH_URL - URL of Flow Runner service.
FLOW_DESIGNER_URL - URL of Designer service.
BASE_DAL_URL - URL of Data Access Layer service.
ENABLE_INSPECT - Default false. If true adds --inspect to entry-point, Debugger will listen on port 9279
ENABLE_ENV_VAR_DEBUG - Default false. If true, then enables to retrieve environmental variables for display in " Service config" tab.
CACHE_SERVICE - URL of Cache service.
ACE_LOG_EXCHANGE - (default: <empty>) - If set to true, logs request(url, method, body) and response(statusCode, body) related data at info level.

Sizing

DEV/TEST (Pods, vCPU, RAM, Disk)	PROD (Pods, vCPU, RAM, Disk)
1, 500m, 256Mi, 5Gb	2, 1, 512Mi, 5Gb

Scaling

This service can be scaled Horizontally, make sure to deploy the same configuration to all instances when scaling.

Data Access Layer

All persistent configiurations are accessed through Data Access Layer.

Container name

flow-enrich-dal

Deployment options

DEV/TEST	PROD
Required	Required

Configuration options

PORT - port used by service.
LOG_LEVEL - Default info. Service log verbosity level. Possible values error, warn, info, http, verbose, debug, silly.
STORAGE_ENV - How to store/read configuration. Currently available option is file.
STORAGE_LOCATION - Storage location on disk. Available if STORAGE_ENV is file. Default value - /usr/src/app/databases
ENABLE_INSPECT - Default false. If true adds --inspect to entry-point, Debugger will listen on port 9269
ENABLE_ENV_VAR_DEBUG - Default false. If true, then enables to retrieve environmental variables for display in " Service config" tab.
ACE_LOG_EXCHANGE - (default: <empty>) - If set to true, logs request(url, method, body) and response(statusCode, body) related data at info level.

Sizing

DEV/TEST (Pods, vCPU, RAM, Disk)	PROD (Pods, vCPU, RAM, Disk)
1, 1, 512Mi, 5Gb	2, 2, 1Gi, 5Gb

Scaling

This service can be scaled Horizontally, make sure to deploy the same configuration to all instances when scaling or when in edit mode, deploy shared volume, where configurations are stored.

Logging

All ACE services are logging according to env variable LOG_LEVEL supplied to the service. It is possible to capture logs in any Log aggregation solution by capturing what is being logged to stdout.

To prevent GDPR related issues sensitive data is not logged by default

User can provide following env variables in 'Settings' related to logs. FULL_DATA_LOGGING = true to enable logging of sensitive data. ACE_LOG_EXCHANGE = true to log request(url, method, body) and response(statusCode, body) related data at info level.

It is also possible to log custom business information using "Logger" step in flow

Security

danger

It is critical to disable Management APIs in production, see Orchestration service configuration

When exposing Dynamic APIs to Consumers use API Gateway or any other infrastructure components to secure * *Orchestration** Dynamic APIs.

Monitoring

There are no considerable warmups livenessProbe and readinessProbe should be defined on per $HOST:$Port basis.

Tracing

caution

Jaeger tracing is removed from version 4.27.0 onwards

ACE currently supports Jaeger distributed tracing for the following services

flow-runner - does not yet support JAEGER_AUTO_INSTRUMENTING
flow-enrich-dal
flow-designer-server
dynamic-api-server
flow-versioning
digital-api-orch

Configuration options available for each service

JAEGER_ENABLED - Default false. If true then enables Jaeger
JAEGER_AUTO_INSTRUMENTING - Default true. If true, creates trace between requests, if false, only notifies on start and end of requests
Further Jaeger configuration

Services​

Flow Manager​

Container name​

Deployment options​

Configuration options​

Sizing​

Scaling​

API Gateway​

Orchestration​

Container name​

Deployment options​

Configuration options​

Sizing​

Scaling​

Designer​

Container name​

Deployment options​

Configuration options​

Sizing​

Scaling​

Dynamic APIs​

Container name​

Deployment options​

Configuration options​

Sizing​

Scaling​

Flow Runner​

Container name​

Deployment options​

Configuration options​

Sizing​

Scaling​

Cache​

Container name​

Deployment options​

Configuration options​

Sizing​

Scaling​

Redis​

Container name​

Deployment options​

Sizing​

Scaling​

Versioning​

Container name​

Deployment options​

Configuration options​

Sizing​

Scaling​

Data Access Layer​

Container name​

Deployment options​

Configuration options​

Sizing​

Scaling​

Logging​

Security​

Monitoring​

Tracing​

Configuration options available for each service​

Services

Flow Manager

Container name

Deployment options

Configuration options

Sizing

Scaling

API Gateway

Orchestration

Container name

Deployment options

Configuration options

Sizing

Scaling

Designer

Container name

Deployment options

Configuration options

Sizing

Scaling

Dynamic APIs

Container name

Deployment options

Configuration options

Sizing

Scaling

Flow Runner

Container name

Deployment options

Configuration options

Sizing

Scaling

Cache

Container name

Deployment options

Configuration options

Sizing

Scaling

Redis

Container name

Deployment options

Sizing

Scaling

Versioning

Container name

Deployment options

Configuration options

Sizing

Scaling

Data Access Layer

Container name

Deployment options

Configuration options

Sizing

Scaling

Logging

Security

Monitoring

Tracing

Configuration options available for each service