One of fundamental principles of programming with Zato is that one's services are typically insulated from inner workings of underlying data formats or security schemes - after all, why bother with mundane tasks such as authentication or authorization, it should be the platform's job whereas user services should rather focus on their own job.

Conversely, once a security definition is created in web-admin, it can be applied in multiple places without requiring any changes to Python code.

All of that applies to a newly added feature of Zato that lets one keep API secrets in Vault and this blog post introduces key concepts behind it.


What is Vault?

Vault is a dedicated tool to store and make use of secrets that offers support for common authentication and authorization related workflows, such as secure storage and broad or fine-grained secrets management as well as policies describing what action a given holder of a secret can perform.

The system is open to customizations and makes for a great companion to Zato in letting one easily express who can access what, for how long, and under what conditions.

Vault needs to be installed separately to Zato. This post assumes that there is a Vault instance running on http://localhost:49517, already unsealed, using the configuration as below.

Essentially, this is a development server but unlike the Vault's default dev server, this one keeps data on disk instead of RAM.

disable_mlock = true

backend "file" {
  path = "./vault-dev.db"

listener "tcp" {
  address = ""
  tls_disable = 1

Creating Vault connections

The easiest way to create a new connection is through web-admin, simply fill out the form as below and a new connection to Vault will be created.


  • Name - an arbitrary name of the connection
  • URL - where Vault can be found
  • Token - Vault token, note that it should be possible for this token to look up tokens and credentials of incoming requests
  • Default authentication method - which authentication backend to use default to authenticate requests, can be Token, Username/password or GitHub. LDAP is coming up soon.
  • Service - a service that can be invoked if no default method was defined - the service can extract credentials from request and indicate what method to use.
  • Timeout - how long to wait for responses from Vault
  • TLS options - whether TLS connections should be verified and if so, using what CA certs unless default ones should be employed. Also, an optional client TLS key and certificate can be uploaded so that Zato itself authenticates with Vault using this key/cert pair.

Attaching Vault connections to channels

Authentication with Vault works with regular HTTP REST or SOAP channels as well as with WebSockets - pick the newly created Vault connection from the list, click OK, and this is it, your channels will now be secured by Vault.

If you have already existing channels, you can swap out their current security definitions for Vault-based ones and things will just work without server restarts.




The above is everything that is needed to define API endpoints backed by Vault which will now on behalf of Zato authenticate users according to what was provided on input when invoking a service.

However, a complementary aspect is that of authorization - i.e., assuming that in the incoming request there were valid credentials and the service can be invoked at all, we can do one better, go a step further, and define Vault policies to express business relations between objects exchanged in API calls. This will let one store rules in a central place accessible to any API endpoint.

This part of the Zato-Vault interface is still under active development - watch this space for more information!


This blog post introduces the Zato scheduler - a feature of Zato that lets one configure API services and background jobs to be run at selected intervals and according to specified execution plans without any programming needed.

Future posts will introduce all the details whereas this one gently describes major features - overall architecture, one-time jobs, interval-based jobs, cron-style jobs and public API to manage jobs from custom tools.


In Zato, scheduler jobs are one of channels that a service can be invoked from - that is, the same service can be mounted on a REST, SOAP, AMQP, ZeroMQ or any other channel in addition to being triggered from the scheduler. No changes in code are required to achieve it.


For instance, let's consider the code below of a hypothetical service that downloads billings from an FTP resource and sends them off to a REST endpoint:

from zato.server.service import Service

class BillingHandler(Service):
    def handle(self):

        # Download data ..
        ftp = self.outgoing.ftp.get('Billings')
        contents = ftp.getcontents('/data/current.csv')

        # .. and send it to its recipient.
        self.outgoing.plain_http['ERP'].post(self.cid, contents)

Nowhere in the service is any reference embedded as to how it will be invoked and this is the crucial part of Zato design - services only focus on doing their jobs not on how to make themselves available from one channel or another.

Thus, even if a service to bulk transfer billings initially will likely be invoked from the scheduler only, there is nothing preventing it from being triggered by a REST call or from command line as needed.

Or perhaps a message sent to an AMQP should trigger it - that is fine as well and the service will not need be changed to accomodate it.

Working with scheduler jobs

There are three types of scheduler jobs:

  • one-time jobs - great if a service should be invoked at a specific time but it does not need to be repeated further
  • interval-based jobs - let one specify how often to invoke a given service (e.g. once in four weeks, twice an hour, five times a minute) as well as when to stop it so as form complex plans such as 'After two weeks from now, invoke this service twice an hour but do it twelve times only'.
  • cron-style jobs - work similar to interval-based ones but use syntax of Cron so 00 3-6 * * 1-5 will mean 'run the service each full hour from 3am to 6am but only Monday to Friday (i.e. excluding weekends)'

Note that job definitions always survive cluster restarts - this means that if you fully shut down a whole cluster of Zato servers then all jobs will continue to execute once the server is back. However, any jobs missed during the downtime will not be re-scheduled.

When a job is being triggered, its target service can receive extra data that may be possibly needed for that service to perform its tasks - this data is completely opaque to Zato and can be in any format, JSON, XML, YAML, plain text, anything.

If a job should not be scheduled anymore - be it because it was a one-time job or because it reached its execution limit, it becomes inactive rather than being deleted.

Such an inactive job still is available in web-admin and can be made active again, possibly with a different schedule plan. On the other hand, actually deleting a job deletes it permanently.





Full public API is available to manage jobs either through REST or SOAP calls as well as from other services directly in Python, such as below:

from zato.common import SCHEDULER

class JobManager(Service):

    def handle(self):

        # Create a sample job that will trigger one of built-in test services

        self.invoke('zato.scheduler.job.create', {
            'cluster_id': self.server.cluster_id,
            'is_active': True,
            'name': 'My Sample',
            'service': 'zato.helpers.input-logger',
            'job_type': SCHEDULER.JOB_TYPE.INTERVAL_BASED,
            'seconds': 2,

Stay tuned for more!

This was just the first installment that introduced core concepts behind the Zato scheduler - coming up are details of how to work with each kind of the jobs, their API and how to efficiently manage their definitions in source code repositories.


In a recent project, an interesting situation occurred that let JSON Web Tokens (JWT) and WebSockets, two newly added features of Zato middleware server, be nicely employed in practice with great results.

The starting point was the architecture as below.


Pretty common stuff - users authenticate with a frontend web-application serving pages to browsers and at the same time communicating with Zato middleware which provides a unified interface to further backend systems.


Now, the scenario started to look intriguing when at one point a business requirement meant that in technical terms it was decided that WebSocket connections be employed so that browsers could be swiftly notified of events taking place in backend systems.

WebSockets are straight-forward, come with all modern browsers, and recently Zato grew means to mount services on WebSocket channels - this in addition to REST, SOAP, ZeroMQ, AMQP, WebSphere MQ, scheduler, and all the other already existing channels.



The gotcha

However, when it came to implementation it turned out that the frontend web-application is incapable to act as a client of Zato services exposed via WebSockets.

That is, it could offer WebSockets to browsers but would not be able itself to establish long-running WebSocket connections to Zato - it had been simply designed to work in a strict request-reply fashion and WebSockets were out of its reach.

This meant that it was not possible for Zato to notify the frontend application of new events without the frontend constantly polling for them which beat the purpose of employing WebSockets in the first place.

Thus, seeing as browsers themselves support WebSockets very well, it was agreed that there is no choice but have each user browser connect to Zato directly and WebSocket channels in Zato would ensure that browsers receive notification as they happen in backend systems.

Browser authentication

Deciding that browsers connect directly to Zato posed a new challenge, however. Whereas previously users authenticated with the frontend that had its own application-level credentials in Zato, now browsers connecting directly to Zato would also have to authenticate.

Naturally, it was ruled out that suddenly users would be burdened with a new username/password to enter anywhere. At the same time it was not desirable to embed the credentials in HTML served to browsers because that would have to be done in clear text.

Instead, JWT was used by the frontend application to securely establish a session in Zato and transfer its ownership to a browser.



How JWT works in Zato

At their core, JWT (JSON Web Tokens) are essentially key-value mappings that declare that certain information is true. In the context of Zato authentication, when selected services Zato server are secured with JWT, the following happens:

  • Applications need to obtain a token for the security definition assigned to the service - they do it either by sending username/password in to an endpoint that returns a new token or by calling a special API service that lets an application generate a token on behalf of another application
  • No matter how it was generated, the token will contain information for whom it was generated and until when it is valid
  • Such a token is next encrypted on server side using Fernet keys (AES-128)
  • After obtaining the token, the end application needs to provide it to Zato on each call
  • When a request comes in, the process is reversed - a JWT is decrypted on server side, its validity confirmed, a service is called and its response is returned to the calling application
  • Since tokens would have expired ultimately otherwise, their validity is extended each time a call is made to Zato with a JWT which indicates to servers that the token is still in use

In other words, JWT declares that a username/password combination was valid at a certain time and that this particular token was generated for that user and that it will be valid until it expires or is further prolonged.

This is very convenient and easy to understand. Another really great property of JWT is that they are extremely simple to use in JavaScript code running in browsers or elsewhere - they are just an opaque string that needs to be provided to Zato servers in a single header which is a trivial task.

Combining it all

Having all the pieces in one place meant the solution was simple - it was the frontend application that would call a custom endpoint in Zato to create a JWT for use in browsers. Since the token is safely encrypted, it can be passed around anywhere and Zato can return it to frontend without any worries.

Once the frontend returns the token to a browser, the browser can then go ahead and open a direct WebSocket connection secured with the newly generated token. Zato receives the token, decrypts it and confirms that it is valid and was in fact generated on server side as expected.

The net result is that browsers now have secure direct WebSocket connections to Zato yet no user credentials are relayed anywhere in clear text.


At the same time, users started to receive notifications from backend systems and everyone was excited even though initially the situation looked bleak when it turned out that the frontend couldn't itself become a WebSockets client.

While it is not very common to connect one's Zato environments to dozens of different AMQP or JMS WebSphere MQ queue managers at a time, one can frequently have dozens or hundreds of services or REST channels declared in a given cluster.

Thus for Zato 3.0 (which is a work in progress) a nice little option was added to paginate web-admin's search results as in the screenshot below:


In fact, all listings will sport this feature in case that one actually does need to connect to dozens of distinct brokers simultaneously :-)

One of many great things about Zato is the fact how easy it is to plug into it new data sources and input methods triggering one's SOA/API services.

For instance, Zato 2.0 does not have a web-admin GUI for file notifications but it is still possible to listen for new or updated files in directories of choice and invoke services each time a new event arrives, e.g. when a new file is dropped into a directory, effectively creating a new channel type in addition to what Zato comes with out of the box.

This is exactly what the script below does - it uses watchmedo, part of the watchdog package, to listen for events in a given directory. Each time anything of interest happens in that directory (here - in /tmp/data) a Zato service defined below is invoked with a path to that item of interest provided on input.


watchmedo shell-command \
    --patterns="*" \
    --command='curl localhost:11223/file.notifications?path=${watch_src_path}; echo' \

It's entirely up to that service to decide how to handle the data in each file - one can decide to deliver it to any of outgoing connections in Zato using, for instance, AMQP, SMTP, FTP, ElasticSearch, Solr or do anything else that is required in a given integration scenario.


Just to exemplify the idea, in this blog post the data is simply stored in server.log:

# -*- coding: utf-8 -*-

from __future__ import absolute_import, division, print_function, unicode_literals

# Zato
from zato.server.service import Service

class FileNotifications(Service):
    name = 'file.notifications'

    class SimpleIO(object):
        input_required = ('path',)

    def handle(self):

        # Read data in
        data = open(self.request.input.path).read().strip()

        # Since this is just an example, only log the contents of what was read in
        self.logger.info('Data from %s is `%s`', self.request.input.path, data)

Such a service needs to be mounted on an HTTP channel, as below.


If you have not done it yet, create a directory /tmp/data and start the watchmedo script above before continuing with the next steps.

Now we can create a new file and observe in server.log what happens, how the service reacts to it:


Sure enough, server.log confirms that everything works as expected: