Overview

This blog post introduces the Zato scheduler - a feature of Zato that lets one configure API services and background jobs to be run at selected intervals and according to specified execution plans without any programming needed.

Future posts will introduce all the details whereas this one gently describes major features - overall architecture, one-time jobs, interval-based jobs, cron-style jobs and public API to manage jobs from custom tools.

Architecture

In Zato, scheduler jobs are one of channels that a service can be invoked from - that is, the same service can be mounted on a REST, SOAP, AMQP, ZeroMQ or any other channel in addition to being triggered from the scheduler. No changes in code are required to achieve it.

Screenshot

For instance, let's consider the code below of a hypothetical service that downloads billings from an FTP resource and sends them off to a REST endpoint:

from zato.server.service import Service

class BillingHandler(Service):
    def handle(self):

        # Download data ..
        ftp = self.outgoing.ftp.get('Billings')
        contents = ftp.getcontents('/data/current.csv')

        # .. and send it to its recipient.
        self.outgoing.plain_http['ERP'].post(self.cid, contents)

Nowhere in the service is any reference embedded as to how it will be invoked and this is the crucial part of Zato design - services only focus on doing their jobs not on how to make themselves available from one channel or another.

Thus, even if a service to bulk transfer billings initially will likely be invoked from the scheduler only, there is nothing preventing it from being triggered by a REST call or from command line as needed.

Or perhaps a message sent to an AMQP should trigger it - that is fine as well and the service will not need be changed to accomodate it.

Working with scheduler jobs

There are three types of scheduler jobs:

  • one-time jobs - great if a service should be invoked at a specific time but it does not need to be repeated further
  • interval-based jobs - let one specify how often to invoke a given service (e.g. once in four weeks, twice an hour, five times a minute) as well as when to stop it so as form complex plans such as 'After two weeks from now, invoke this service twice an hour but do it twelve times only'.
  • cron-style jobs - work similar to interval-based ones but use syntax of Cron so 00 3-6 * * 1-5 will mean 'run the service each full hour from 3am to 6am but only Monday to Friday (i.e. excluding weekends)'

Note that job definitions always survive cluster restarts - this means that if you fully shut down a whole cluster of Zato servers then all jobs will continue to execute once the server is back. However, any jobs missed during the downtime will not be re-scheduled.

When a job is being triggered, its target service can receive extra data that may be possibly needed for that service to perform its tasks - this data is completely opaque to Zato and can be in any format, JSON, XML, YAML, plain text, anything.

If a job should not be scheduled anymore - be it because it was a one-time job or because it reached its execution limit, it becomes inactive rather than being deleted.

Such an inactive job still is available in web-admin and can be made active again, possibly with a different schedule plan. On the other hand, actually deleting a job deletes it permanently.

Screenshot

Screenshot

Screenshot

Screenshot

Full public API is available to manage jobs either through REST or SOAP calls as well as from other services directly in Python, such as below:

from zato.common import SCHEDULER

class JobManager(Service):

    def handle(self):

        # Create a sample job that will trigger one of built-in test services

        self.invoke('zato.scheduler.job.create', {
            'cluster_id': self.server.cluster_id,
            'is_active': True,
            'name': 'My Sample',
            'service': 'zato.helpers.input-logger',
            'job_type': SCHEDULER.JOB_TYPE.INTERVAL_BASED,
            'seconds': 2,
        })

Stay tuned for more!

This was just the first installment that introduced core concepts behind the Zato scheduler - coming up are details of how to work with each kind of the jobs, their API and how to efficiently manage their definitions in source code repositories.

Overview

In a recent project, an interesting situation occurred that let JSON Web Tokens (JWT) and WebSockets, two newly added features of Zato middleware server, be nicely employed in practice with great results.

The starting point was the architecture as below.

Screenshot

Pretty common stuff - users authenticate with a frontend web-application serving pages to browsers and at the same time communicating with Zato middleware which provides a unified interface to further backend systems.

WebSockets

Now, the scenario started to look intriguing when at one point a business requirement meant that in technical terms it was decided that WebSocket connections be employed so that browsers could be swiftly notified of events taking place in backend systems.

WebSockets are straight-forward, come with all modern browsers, and recently Zato grew means to mount services on WebSocket channels - this in addition to REST, SOAP, ZeroMQ, AMQP, WebSphere MQ, scheduler, and all the other already existing channels.

Screenshot

Screenshot

The gotcha

However, when it came to implementation it turned out that the frontend web-application is incapable to act as a client of Zato services exposed via WebSockets.

That is, it could offer WebSockets to browsers but would not be able itself to establish long-running WebSocket connections to Zato - it had been simply designed to work in a strict request-reply fashion and WebSockets were out of its reach.

This meant that it was not possible for Zato to notify the frontend application of new events without the frontend constantly polling for them which beat the purpose of employing WebSockets in the first place.

Thus, seeing as browsers themselves support WebSockets very well, it was agreed that there is no choice but have each user browser connect to Zato directly and WebSocket channels in Zato would ensure that browsers receive notification as they happen in backend systems.

Browser authentication

Deciding that browsers connect directly to Zato posed a new challenge, however. Whereas previously users authenticated with the frontend that had its own application-level credentials in Zato, now browsers connecting directly to Zato would also have to authenticate.

Naturally, it was ruled out that suddenly users would be burdened with a new username/password to enter anywhere. At the same time it was not desirable to embed the credentials in HTML served to browsers because that would have to be done in clear text.

Instead, JWT was used by the frontend application to securely establish a session in Zato and transfer its ownership to a browser.

Screenshot

Screenshot

How JWT works in Zato

At their core, JWT (JSON Web Tokens) are essentially key-value mappings that declare that certain information is true. In the context of Zato authentication, when selected services Zato server are secured with JWT, the following happens:

  • Applications need to obtain a token for the security definition assigned to the service - they do it either by sending username/password in to an endpoint that returns a new token or by calling a special API service that lets an application generate a token on behalf of another application
  • No matter how it was generated, the token will contain information for whom it was generated and until when it is valid
  • Such a token is next encrypted on server side using Fernet keys (AES-128)
  • After obtaining the token, the end application needs to provide it to Zato on each call
  • When a request comes in, the process is reversed - a JWT is decrypted on server side, its validity confirmed, a service is called and its response is returned to the calling application
  • Since tokens would have expired ultimately otherwise, their validity is extended each time a call is made to Zato with a JWT which indicates to servers that the token is still in use

In other words, JWT declares that a username/password combination was valid at a certain time and that this particular token was generated for that user and that it will be valid until it expires or is further prolonged.

This is very convenient and easy to understand. Another really great property of JWT is that they are extremely simple to use in JavaScript code running in browsers or elsewhere - they are just an opaque string that needs to be provided to Zato servers in a single header which is a trivial task.

Combining it all

Having all the pieces in one place meant the solution was simple - it was the frontend application that would call a custom endpoint in Zato to create a JWT for use in browsers. Since the token is safely encrypted, it can be passed around anywhere and Zato can return it to frontend without any worries.

Once the frontend returns the token to a browser, the browser can then go ahead and open a direct WebSocket connection secured with the newly generated token. Zato receives the token, decrypts it and confirms that it is valid and was in fact generated on server side as expected.

The net result is that browsers now have secure direct WebSocket connections to Zato yet no user credentials are relayed anywhere in clear text.

Screenshot

At the same time, users started to receive notifications from backend systems and everyone was excited even though initially the situation looked bleak when it turned out that the frontend couldn't itself become a WebSockets client.

While it is not very common to connect one's Zato environments to dozens of different AMQP or JMS WebSphere MQ queue managers at a time, one can frequently have dozens or hundreds of services or REST channels declared in a given cluster.

Thus for Zato 3.0 (which is a work in progress) a nice little option was added to paginate web-admin's search results as in the screenshot below:

Screenshots

In fact, all listings will sport this feature in case that one actually does need to connect to dozens of distinct brokers simultaneously :-)

One of many great things about Zato is the fact how easy it is to plug into it new data sources and input methods triggering one's SOA/API services.

For instance, Zato 2.0 does not have a web-admin GUI for file notifications but it is still possible to listen for new or updated files in directories of choice and invoke services each time a new event arrives, e.g. when a new file is dropped into a directory, effectively creating a new channel type in addition to what Zato comes with out of the box.

This is exactly what the script below does - it uses watchmedo, part of the watchdog package, to listen for events in a given directory. Each time anything of interest happens in that directory (here - in /tmp/data) a Zato service defined below is invoked with a path to that item of interest provided on input.

#!/bin/bash

watchmedo shell-command \
    --patterns="*" \
    --command='curl localhost:11223/file.notifications?path=${watch_src_path}; echo' \
    /tmp/data

It's entirely up to that service to decide how to handle the data in each file - one can decide to deliver it to any of outgoing connections in Zato using, for instance, AMQP, SMTP, FTP, ElasticSearch, Solr or do anything else that is required in a given integration scenario.

Screenshots

Just to exemplify the idea, in this blog post the data is simply stored in server.log:

# -*- coding: utf-8 -*-

from __future__ import absolute_import, division, print_function, unicode_literals

# Zato
from zato.server.service import Service

class FileNotifications(Service):
    name = 'file.notifications'

    class SimpleIO(object):
        input_required = ('path',)

    def handle(self):

        # Read data in
        data = open(self.request.input.path).read().strip()

        # Since this is just an example, only log the contents of what was read in
        self.logger.info('Data from %s is `%s`', self.request.input.path, data)

Such a service needs to be mounted on an HTTP channel, as below.

Screenshots

If you have not done it yet, create a directory /tmp/data and start the watchmedo script above before continuing with the next steps.

Now we can create a new file and observe in server.log what happens, how the service reacts to it:

Screenshots

Sure enough, server.log confirms that everything works as expected:

Screenshots

As of today, the mailing list used by Zato has been retired in favour of the new forum that will let us provide a great community experience.

All of the contents of the mailing list has been already migrated to the forum so all the terrific questions, answers and ideas posted to the list are still accessible in the new forum.

Speaking of the migration - a Python tool has been written to automate the process and its source code is published on GitHub. This can be taken and forked at will with the hope that it can serve as a source of inspiration on how to use the Discourse API from Python code.