zato enmasse
Overview
Enmasse is a command line tool that automates the deployment of Zato configuration, such as REST or SQL connections, scheduler and security definitions or any other object.
During development, one usually uses the Dashboard GUI to define new elements, e.g. it is usually convenient to fill out a form to have a new REST channel available.
When a solution, or an incremental part of it, is ready for wider testing, enmasse is used to export it from the development server and to import it to a test server.
Afterwards, the same enmasse export file can be used to configure production servers.
The workflow loop now returns to developers who keep using Dashboard to add new server elements
All of the export and import steps can be part of CI/CD pipelines that automate the provisioning of Zato environments
Anything that can be declared with the Dashboard GUI can be also done with enmasse. For instance, in the screenshot below a new REST channel is being defined.
Now, enmasse can be used to export such a channel to YAML and you will be able to import it later on.
Such export files can be stored in a git repository. Iteratively, developers add new definitions to the file based on the current state of their work. Ultimately, the file will contain each definition, e.g. each REST channel and other connection types, that are needed to configure a particular project or solution.
To prevent any potential leakage of sensitive information, enmasse will not export any passwords from servers to the YAML file. Yet, it is always possible to add passwords to your enmasse files via environment variables, as described later in this chapter, so that you will be still able to recreate fully each definition with passwords where they are needed.
In a multi-server cluster, it suffices for one server to import an enmasse file. It will synchronize automatically with all the other servers in the cluster.
You can run enmasse from the same operating system that your server is in, e.g. from the same Linux VM or container, or you can use enmasse in combination with Docker Quickstart.
An enmasse file contains configuration only, it does not contain services that the configuration may depend on. Any such services need to be deployed before enmasse imports your configuration file.
You do not need to create an object first in Dashboard before it can be added to an enmasse file. For instance, you may have many similar REST channels or security definitions, differing only in one or two small details, and it will be more convenient if you create one in Dashboard, export it so that you can learn what its format in enmasse looks like, and then create the remaining definitions in the enmasse file directly. In other words, after a while, you will become familiar with enmasse to the point that you will be adding definitions directly to it.
How to export and import definitions
Exporting configuration to a file called "enmasse.yaml":
$ zato enmasse --export --output enmasse.yaml /path/to/server/
[...]
Data exported to /opt/zato/enmasse.yaml
$
Importing configuration from a file called "enmasse.yaml":
Environment variables
When enmasse runs, it can read environment variables and replace selected parts of a file to be imported based on what is found in the environment.
Passwords and other types of credentials are what environment variables are usually used for but it is possible to use them with any values in enmasse files.
To provide a value for the whole key based on what is among environment variables, give prefix "Zato_Enmasse_Env" to such a variable, as in the example below which reads a password for this security definition from a variable called "My_Token_Password" which must exist in the same system where enmasse runs.
security:
- name: My.Token
username: api
type: bearer_token
password: Zato_Enmasse_Env.My_Token_Password
To replace a part of a value, use the syntax below, which involves placing the same prefix as above inside curly brackets-delimited placeholders. You can have more than one placeholder in the same value.
In this example, if there is an environment variable called "My_Name_Suffix" and its name is "API.REST", the final name of this REST channel will be "My.API.REST".
Environment variables are read from the standard places, such as ~/.bashrc or from what was exported to a starting Docker container. It is also possible to read them from a file through an option called "--env-file", as in the example below.
The file has the following format. There is a section called "[env]" which contains a list of environment variables, each key/value pair in its own line. From enmasse's perspective, it does not matter if a variable was exported through ~/.bashrc, Docker variables or via such a file.
Including other enmasse files
At times, it may be convenient to break a single enmasse file into more than one. For instance, common security definitions may be kept in a single file that multiple deployment-specific enmasse files will include. Or you can keep more than one enmasse file depending on which project you are going to deploy.
To include other enmasse files, use the syntax below:
include:
- relative_name1.yaml
- ../relative_name2.yaml
- /absolute_name1.yaml
- /absolute/name2.yaml
The name of an enmasse file to include may be relative or absolute. If it is relative, it will be resolved in relation to the directory in which your main enmasse file is.
There can be at most one include section in an enmasse file and it does not matter where in the enmasse file the include section is placed although it is recommended that it be placed at the very beginning of the file.
All includes are resolved and included first, before any other definitions from a given file are read. This lets you keep common dependencies, such as security definitions, in one file that is included from more than one enmasse file.
Files that you include may include other files. This process will repeat until all files are included and a final list of definitions is obtained, at which point enmasse will run the import process.
Do not include files recursively or mutually, i.e. do not include a file in itself and, for instance, do not let file "enmasse1.yaml" include file "enmasse2.yaml" if "enmasse2.yaml" directly or indirectly includes "enmasse1.yaml" back.
If needed, use the "--ignore-missing-includes" option to ignore include files that do not exist when running an import. This is useful if your include files are optional depending on runtime circumstances and you do not want for enmasse to report an error if some of them are missing.
Choosing what to export
You can choose to export only selected objects from a server to your enmasse file by filtering by their names or types. Filtering is case-insensitive.
For instance, this will export only these objects whose name contains the string "billing", regardless of their type, e.g. it does not matter if the object is a REST channel, an LDAP connection, or any other:
This sample will export only objects with names matching "billing" or "crm":
Using "--include-name all" is the same as not providing the "--include-name" option and will disable filtering by name.
In a similar way, you can choose to export only selected types of objects. You can choose one or more of "cache", "cloud-microsoft-365", "rest", "security" "sql", "scheduler", "pubsub", "ldap", or "all". If you do not use this option, it will be the same as choosing "all". Note that filtering by type is possible for the types that are documented here, e.g. it is not enabled for "amqp", "imap" or other types of connections.
For instance, to export all the REST definitions, regardless of their name or other attributes:
This example will export all REST and scheduler definitions, no matter what their name is:
You can combine "--include-type" and "--include-name". As an example, to export scheduler's definitions but only these that have "billing" in their names:
If you provide both "--include-type" and "--include-name", and they are a list of values, they will be or-joined.
For instance, this sample will export all scheduler and REST definitions whose name contains strings "billing" or "crm". That is, it will export all scheduler definitions that have "billing" or "crm" in their name as well as all the REST definitions that have "billing" or "crm" in their name.
Choosing what attributes to import
The most commonly used object types, such as REST, security or SQL definitions, are exported by enmasse using as few attributes as possible.
For instance, even if REST channels can potentially be described by many attributes, most installations will use the default values which is why, unless the default values have been actually overridden before an export, enmasse exports such channels in a concise format, as below:
To understand what other attributes can be imported for a given object type, do this:
Run
tail -fon the "admin.log" file which each server has, e.g. "tail -f /path/to/server/logs/admin.log". This file records all the administration and configuration requests and responses that a server receives in a JSON format.Go to your Dashboard and create or edit a given object, such as a REST channel or anything else.
Observe in the "admin.log" file the request that Dashboard sent to the server. The request will contain all the possible attributes that can be set for this object, including empty or default ones that enmasse does not export yet you will be able to import them if you add them to your enmasse file.
Add any required attributes to the enmasse file based on what you have found in "admin.log". For instance, by comparing what is available in Dashboard with the "admin.log" file, you may learn that REST channels have an attribute called "method", which you will be able to set as below.
Running enmasse when a server is not available
Depending on how your provisioning infrastructure is set up, you may not always know if, at the time when enmasse runs, the server it is trying to connect to is already running and, if it is running, if it has not potentially just started and has not deployed all the services yet.
If you run enmasse, and it discovers that the server is not running at all, it will wait for it for up to 12 hours during which time it will periodically try to connect to the server - note that it is 12 hours, not seconds or minutes. This timeout is used by both export and import actions.
You can override the initial wait time by using the "--initial-wait-time" parameter. In the example below, it is set to half an hour (1800 seconds):
Additionally, when a server is starting up, it will typically require a few or more seconds to deploy all of its services. It means that, even if it responding, and enmasse will notice that it is up, it may still not be ready for enmasse to run because some of the services are not deployed yet.
If some of the services required by definitions that enmasse is trying to import are missing in a server, enmasse will keep trying to run the import periodically for a total of 120 seconds. If this limit is reached, enmasse will exit with an error. Note that this timeout applies only to the import action.
To change the timeout of how many seconds to wait for missing services:
Ignoring missing enmasse files
In certain scenarios, it is desirable to ignore enmasse files that do not exist instead of having Zato report an error. This can be useful if your enmasse files are potentially available in multiple locations and, depending on circumstances, sometimes they may not exist in some of them.
For instance, instead of exiting with a status code indicating an error, this command will exit with the status code of 0, indicating success, if the input "enmasse.yaml" file is not found. If the file is found, it will be imported as usual.
Ignoring missing include files
In certain situations, it is preferred to ignore missing include files that other enmasse files should incorporate. Use the "--ignore-missing-ignores" option to achieve that.
Read more
A separate chapter discusses how to deploy services and other code