Configuring a server

Purpose File Example full path
Main config server.conf /opt/zato/dev1/server1/config/repo/server.conf
Logging configuration logging.conf /opt/zato/dev1/server1/config/repo/logging.conf
Service sources service-sources.txt /opt/zato/dev1/server1/config/repo/service-sources.txt
Crypto material zato-server-*.pem /opt/zato/dev1/server1/config/repo/zato-server-*.pem

Main config - server.conf

Discussion

main.gunicorn_bind

Host and port to bind to. Note that it always needs to synchronized with the address the LB will attempt to find a server at.

main.gunicorn_worker_class

Main worker class - using ‘gevent’, the default value makes Zato servers non-blocking asynchronous ones however advanced users may wish to use any other value gunicorn support

main.gunicorn_workers

How many worker processes this server should start - in general, the total number of workers in a given system across all servers shouldn’t be more than 2 * CPU_COUNT, so if there are 4 CPUs and 2 servers, each shouldn’t have more than 4 main.gunicorn_workers.

Note that this is a recommendation only, each particular workload may need different settings.

main.gunicorn_timeout

After how many seconds a worker should be considered inactive - this should be set to at least as many seconds as you expect your slowest service to respond in.

For instance, you know each service running on the server will respond under 10 s so it’s safe to set main.gunicorn_timeout to 20 in under to kill any processes with services that take too long to complete.

Likewise, if you know there will be long-running scheduler-initiated batch processes and each needs 60 s to complete, main.gunicorn_timeout should be set to a higher value to allow for a long-running service to complete.

main.gunicorn_user

Same as gunicorn’s user.

main.gunicorn_group

Same as gunicorn’s group.

main.gunicorn_proc_name

Same as gunicorn’s proc_name.

main.gunicorn_logger_class

Same as gunicorn’s logger_class.

main.deployment_lock_expires

When using a disitributed lock to coordinate a hot-deployment of a service, how many seconds to wait for a server to finish the deployment.

Should be at least as many seconds as it takes to hot-deploy a service in the worst case.

main.deployment_lock_timeout

When hot-deploying a service only one server will be allowed to actually perform the action. Other servers will wait up that many seconds for their try to deploy the service in case the server which was previously attempting to hot-deploy it gave up for any reason.

main.token

A unique token automatically assigned to this server purely for the purpose of identyfying a server in the ODB prior to obtaining the server’s ID in the database, i.e. this is akin to a primary key in its purpose.

Must not be repeated across all servers across all cluster re-using the same ODB.

The default value is a UUID4 and it’s recommended that it not be changed.

main.service_sources

Path to the file containing service sources that describe which services to import when the server is starting up.

If not an absolute path, the path is relative to the main server.conf config file.

crypto.use_tls

If True, the server will listen for connections from the cluster’s load-balancer on a SSL/TLS socket instead of a plain TCP one. Refer to a dedicated chapter for details.

crypto.tls_protocol

If crypto.use_tls is True, the SSL/TLS protocol to use. Must be one of:

  • TLSv1
  • SSLv3
  • SSLv23

Defaults to TLSv1 and using other values is highly discouraged.

crypto.tls_ciphers

If crypto.use_tls is True, ciphers the server will support, in OpenSSL’s format.

crypto.tls_client_certs

One of paths to the server’s crypto material. If not absolute, the path is relative to the main config file.

crypto.pub_key_location

One of paths to the server’s crypto material. If not absolute, the path is relative to the main config file.

crypto.cert_location

One of paths to the server’s crypto material. If not absolute, the path is relative to the main config file.

crypto.ca_certs_location

One of paths to the server’s crypto material. If not absolute, the path is relative to the main config file.

odb.db_name

ODB database name.

odb.engine

ODB type, may be one of:

  • mysql
  • postgresql
  • sqlite

odb.extra

Extra parameters passed as kwargs directly into SQLAlchemy’s engine initializer. For instance, setting the option to

[odb]
extra=echo=True;logging_name=foobar

will make SQLAlchemy log all the SQL statements issued using the log name provided.

odb.host

ODB database host.

odb.password

The ODB user’s password to use, encrypted using the server’s private key. The password can be decrypted and encrypted from command line using the CLI.

odb.pool_size

ODB connection pool size.

odb.username

Username to connect to the ODB with.

hot_deploy.pickup_dir

A path to the directory that will be used to hot-deploy Zato services from. If not absolute, the path is relative to the main server.conf config file.

hot_deploy.work_dir

A path to the directory that will be used to:

If not absolute, the path is relative to the main server.conf config file.

hot_deploy.backup_history

Before hot-deploying a service, a backup of already existing services is created. This setting controls how many backups to keep.

hot_deploy.backup_format

Backed-up services are compressed using this format which may be one of the values Python’s shutil.make_archive accepts.

hot_deploy.delete_after_pick_up

Whether Python code should be deleted after it’s been deployed through a pickup dir. It’s convenient to set it to False at least during development as it will make Zato deploy a service module being developed after each save.

This can be used in conjunction with hot_deploy.pickup_dir in order to make hot-deployment work directly from code repository clone.

For instance, setting hot_deploy.pickup_dir to a directory of services under a GitHub clone and configuring hot_deploy.delete_after_pick_up to False means that each time git pull receives new services they will be automatically deployed onto all servers in a cluster.

hot_deploy.current_work_dir

A path to the directory that will be used to hot-deploy the services to.

Must be relative to hot_deploy.work_dir.

hot_deploy.backup_work_dir

A path to the directory that will be used to keep a linear log of service backups.

Must be relative to hot_deploy.work_dir.

hot_deploy.last_backup_work_dir

A path to the directory that will be used to keep the last backup only - this is in addition to the fact that the backup will be stored in hot_deploy.backup_work_dir as well.

deploy_patterns_allowed.*

Which services, including Zato’s own internal ones, can be deployed on the server. Uses patterns documented in their own chapter.

invoke_patterns_allowed.*

Which services, including Zato’s own internal ones, can be invoked on the server. Uses patterns documented in their own chapter.

invoke_target_patterns_allowed.*

Which targets can be executed using invoke or invoke_async on the server. Uses patterns documented in their own chapter.

If a service uses a target name, the request will be executed on a server only if the server supports the target, that is, if the target provided by the service matches one of the patterns configured for server.

For invoke - a service will receive a ZatoException if the target is not supported. For invoke_async, the request will be dropped. Unless on DEBUG level or more verbose the request will be dropped silently.

spring.context_class

Name of a Python class to create the Spring Python context from.

misc.internal_services_may_be_deleted

If a request to delete a Zato’s own internal service arrives, whether this server should allow for it. The value must be consistent across all servers in a given cluster.

misc.return_internal_objects

When returning listings of objects, such as services, whether Zato’s own internal objects should be included in results.

misc.initial_cluster_name

Name of the cluster this server was initially a part of. Useful if a cluster is renamed and there’s a need to look up the server’s original cluster name.

misc.initial_server_name

Original name of this server at the time when it was created.

misc.queue_build_cap

How many seconds queue-based connections will wait for all the connections to be established.

misc.http_proxy

HTTP proxy to use in outgoing connections.

misc.locale

Server’s locale, such as sv_SE.UTF-8. Default value is inherited from environment.

If not empty, in order to confirm correctness of the setting, a starting server will print out the value of 123456 in the locale’s currency.

For instance, if set to de_DE.utf8:

1
INFO - Locale is `de_DE.utf8`, amount of 12345 -> `12.345,00 €`

Yet when set to ja_JP.UTF-8

1
INFO - Locale is `ja_JP.UTF-8`, amount of 12345 -> `¥12,345`

misc.ensure_sql_connections_exist

Whether to perform background pings, several seconds apart, of SQL connections in order to make sure they are still alive. In particular, useful with MySQL which closes connections after a period of inactivity.

misc.http_server_header

Value of the HTTP Server header sent in responses.

$ curl -v localhost:11223/zato/ping
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 11223 (#0)
> GET /zato/ping HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:11223
> Accept: */*
>
< HTTP/1.1 200 OK
* Server Zato is not blacklisted
< Server: Zato
< Date: Wed, 07 Jan 2015 17:59:43 GMT
< Connection: close
< Transfer-Encoding: chunked
< Content-Type: application/json
< X-Zato-CID: K06WFRPSGTDF4Z9CVGNZBDFR5TCF
<
* Closing connection 0
{"zato_env": {"details": "", "result": "ZATO_OK",
  "cid": "K06WFRPSGTDF4Z9CVGNZBDFR5TCF"}, "zato_ping_response":
  {"pong": "zato"}}
$

misc.zeromq_connect_sleep

How long to sleep, in seconds, after initially opening ZeroMQ sockets in messages sent out through outgoing connections.

misc.aws_host

In Amazon S3 connections, host to connect to, such as eu-central-1 - may be required if connecting to region Frankfurt. Applies to all S3 connections.

misc.use_soap_envelope

In SOAP channels, whether responses Zato produces should be wrapped in the SOAP Envelope element or not. Applies to all SOAP channels.

stats.expire_after

At most how many hours of statistics to keep, excluding aggregated ones. Defaults to 1 week (168 hours). Required in order to make sure statistics-related keys in Redis won’t grow indefinitely.

kvdb.*

Settings allowing to connect to the Redis key/value DB which is used as the Zato broker. Note that password, if any should be decrypted and encrypted from command line using the CLI.

  • use_redis_sentinels indicates whether to use Redis HA sentinels instead of connecting to nodes directly
  • redis_sentinels is a list of IP:PORT entries pointing to Redis HA sentinels
  • redis_sentinels_master - name of the master node to connect to
  • shadow_password_in_logs - when logging Redis configuration, whether to shadow out password to connect with
  • log_connection_info_sleep_time - Redis connections print out their configuration to logs that many seconds apart

startup_services_first_worker.*

Each entry is a service, along with its potentially empty input (self.request_raw_request), that will be invoked after it is certain that the first gunicorn worker of the server is ready and fully initialized.

Input can be a string, a list of values separated by commas, or if prefixed with file:// - a path to a file whose contents will be provided to the service on input.

This lets one run startup code only if the current worker is the first one in a given server.

startup_services_any_worker.*

Each entry is a service, along with its potentially empty input (self.request_raw_request), that will be invoked after it is certain that the first gunicorn worker of the server is ready and fully initialized.

Input can be a string, a list of values separated by commas, or if prefixed with file:// - a path to a file whose contents will be provided to the service on input.

This lets one run startup code by each of a server’s gunicorn workers.

pubsub.move_to_target_queues_interval

After receiving a publish message from a producer at least that many seconds will elapse until it will be placed on subscriber queues.

pubsub.delete_expired_interval

After how many seconds to assume that messages with no subscribers can be deleted.

pubsub.invoke_callbacks_interval

How often, in seconds, to attempt to deliver messages to callback-based subscribers.

profiler.*

repoze.profiler-based profiler configuration - consult the underlying implementation’s documentation for details on configuration.

user_config.*

Each entry is a name of a config file while values are paths, possibly relative to the directory the server.conf file is in, containing INI-style configuration for services to use in runtime.

newrelic.*

A set of arguments for integration with NewRelic, passed directly to NewRelic agent’s initialize function.

sentry.*

A set of arguments for integration with Sentry for the Sentry client to use.

rbac.custom_auth_list_service

Name of a user-defined service that will return a list of applications from arbitrary data sources to be consulted in establishing RBAC user permissions.

It’s meant to be used if credentials are kept in a data store to which there is no built-in Zato adapter.

The list must contain dictionaries, each with two keys - client_name and client_def - such as follows:

[
 {'client_name':'My Client1', 'client_def':'USER:::AUTH_TYPE:::APP_NAME1'},
 {'client_name':'My Client2', 'client_def':'USER:::AUTH_TYPE:::APP_NAME2'},
]

Where:

  • client_name - user-friendly of the application the credentials are for
  • client_def - a triple of an arbitrary user-specific tag, authentication type and application code separated by ‘:::’ (three colons).

For instance, if a user’s name is MyCompany, applications are read from DB2 on z/OS and their names are CRM and Billing, the list could look like:

[
 {'client_name':'CRM', 'client_def':'MYCOMP:::DB2ZOS:::CRM'},
 {'client_name':'BILLING', 'client_def':'MYCOMP:::DB2ZOS:::BILLING'},
]

component_enabled.*

Selectively turns off certain Zato compomenets. Currently supported keys are:

Logging configuration - logging.conf

Format

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
loggers:
    '':
        level: INFO
        handlers: [stdout, default]
    zato:
        level: INFO
        handlers: [stdout, default]
        qualname: zato
        propagate: false
    zato_access_log:
        level: INFO
        handlers: [http_access_log]
        qualname: zato_access_log
        propagate: false

Discussion

Uses Python’s own syntax to configure how and where logging messages should be written to.

Of special interest are keys:

  • ‘’.level - sets the log level for anything but Zato
  • zato.level - sets the log level for main Zato logger

Log files

Name Notes
admin.log I/O of admin services used for reconfiguration of running servers
http_access.log An Apache-compatible HTTP access log
kvdb.log Messages related to key/value DB (Redis)
pubsub.log Messages related to publish/subscribe
pubsub-overflown.log Publish/subscribe messages that could not be placed on topics because they were full
rbac.log Messages related to Role-Based Access Control (RBAC)
scheduler.log Messages related to the scheduler
server.log Main server log where user services store information using self.logger

Using alternative loggers

By default logs are written out to local files but it’s possible to reconfigure the logging to use any other destination Python itself supports such as syslog or ZeroMQ.

Service sources - service-sources.txt

Example

1
2
3
4
5
./my-services/import1
./my-services/import2

# This should be kept at the end
./work/hot-deploy/current

Discussion

A list of paths, each relative to the server root, that will be scanned in order to find services to deploy when the server is starting.

Your own services, unless hot-deployed, should be kept in directories each of which has its own entry in this file.

Any line starting with the # sign is ignored.

Crypto material - zato-server-*.pem

Server’s cryptograpical material is kept in PEM files. Private files must not be password-protected.

  • zato-server-priv-key.pem - private key
  • zato-server-pub-key.pem - public key
  • zato-server-cert.pem - certificate
  • zato-server-ca-certs.pem - a list of CA certificates this server should trust

Changelog

Version Notes
2.0

Added new entries:

  • crypto.use_tls
  • crypto.tls_protocol
  • crypto.tls_ciphers
  • misc.return_internal_objects
  • misc.queue_build_cap
  • misc.http_proxy
  • misc.locale
  • misc.ensure_sql_connections_exist
  • misc.http_server_header
  • misc.zeromq_connect_sleep
  • kvdb.use_redis_sentinels
  • kvdb.redis_sentinels
  • kvdb.redis_sentinels_master
  • kvdb.shadow_password_in_logs
  • kvdb.log_connection_info_sleep_time
  • startup_services_first_worker.*
  • startup_services_any_worker.*
  • pubsub.*
  • profiler.*
  • user_config.*
  • newrelic.*
  • sentry.*
  • rbac.*

Added new odb.engine values: mysql and sqlite

Logging configuration’s format changed from INI to YAML

1.0 Added initially