A recurring need in larger integration projects is generation of API documentation for users belonging to different, yet related, target groups. Read on to learn how to generate Zato-based API specifications for more than one group from a single source of information.

A typical scenario is granting access to the same APIs to external and internal users - what they have in common is that all of them may want to access the same APIs yet not all of them should have access to documentation on the same level of details.

For instance - external developers should only know what a given endpoint is for and how to use it but internal ones may also be given information about its inner workings, the kind of details that external users should never learn about.

If documentation for two such groups is kept separately, it may require more maintenance effort than necessary - after all, if most of it is the same for everyone then it would make sense to keep it in one place and only add or remove details, depending on which group particular API documentation needs to be generated for.

The previous article went through the process of generating API specifications step by step - this one goes even further by adding introducing the notion of tags that drive which parts of documentation to generate or not.

Python docstrings

Let us use one of the same services as previously - here is its basic form:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Int, Service

class RechargeCard(Service):
    """ Recharges a pre-paid card.
    Amount must not be less than 1 and it cannot be greater than 10000.
    """

    class SimpleIO:
        input_required = 'number', Int('amount')
        output_required = Int('status')

The result, when generated as Sphinx, will be like below:

Now, we would like to add some details that only internal users should have access to - this is how the docstring should be modified, assuming for a moment that there are two CRM systems in the company the usage of which depends on a particular end user's account ..

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Int, Service

class RechargeCard(Service):
    """ Recharges a pre-paid card.
    Amount must not be less than 1 and it cannot be greater than 10000.

    #internal

    For user accounts starting with QM - MyCRM is queried.
    For any other account - MyCRM-2 is queried.
    """

    class SimpleIO:
        input_required = 'number', Int('amount')
        output_required = Int('status')

.. with the corresponding change in the output:

What happened above was that we applied a tag called #internal - anything that follows it is included in the output only if that very tag is requested when generating the documentation.

Note that the tag name is arbitrary, it could be any other name. Note also that there may be more than tag in a docstring, e.g. #confidential, #private or anything else.

Command-line usage

To actually include the #internal tag, the zato apispec command needs to be told about it explicitly, as below:

$ zato apispec /path/to/server \
  --dir /path/to/output/directory \
  --include api.* \
  --tags public,internal

If you do not give the command any tags, only the public part of the docstring, one without any tags, will be included in the resulting documentation.

To make your clear to other developers, you can also directly use the tag public in the command - this will make it easier to understand that you want to generate publicly available information and nothing else.

$ zato apispec /path/to/server \
  --dir /path/to/output/directory \
  --include api.* \
  --tags public

Wrapping up

This is it, you are done now - you can keep your API documentation in one place now and include only the relevant parts of it depending on context - this ensures that no matter who the recipient of your documentation is, it will be always generated from a single source, thus ensuring that all the output will stay in sync.

In addition to a GUI, Python and REST APIs, it is now possible to access your Zato caches from command line. Learn from this article how to quickly check, set and delete keys in this way - particularly useful for remote SSH connections to Zato environments.

Prerequisites

This functionality will be released in Zato 3.2 (June 2020) - right now, if you would like to use it, Zato needs to be installed from source.

In web-admin

First, let us create a couple of new keys in the GUI - my.key and my.key2 - to work with them later on from command line.

Command line

Now, we can get, set and delete the keys using the CLI. Observe the below and notice that set and delete commands not only carry out what they ought to but they also return the previous value of a given key.

$ zato cache get my.key --path /path/to/server1 ; echo
{"value": "my.value"}
$
$ zato cache get my.key2 --path /path/to/server1 ; echo
{"value": "my.value2"}
$
$ zato cache set my.key my.new.value --path /path/to/server1 ; echo
{"prev_value": "my.value"}
$
$ zato cache delete my.key2 --path /path/to/server1 ; echo
{"prev_value": "my.value2"}
$ zato cache set my.key3 my.value3 --path /path/to/server1 ; echo
{}
$

Back to web-admin

The last command created a new key - we can confirm its existence in web-admin:

Summary

That it is all - as simple as possible, just log in to an SSH server, point your command line to Zato and you can access your caches right away.

This article presents a workflow for auto-generation of API specifications for your Zato services - if you need to share your APIs with partners, external or internal, this is how it can be done.

Sample services

Let's consider the services below - they represent a subset of a hypothetical API of a telecommunication company. In this case, they are to do with pre-paid cards. Deploy them on your servers in a module called api.py.

Note that their implementation is omitted, we only deal with their I/O, as it is expressed using SimpleIO.

What we would like to have, and what we will achieve here, is a website with static HTML describing the services in terms of a formal API specification.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Int, Service

# #####################################################################

class RechargeCard(Service):
    """ Recharges a pre-paid card.
    Amount must not be less than 1 and it cannot be greater than 10000.
    """
    class SimpleIO:
        input_required = 'number', Int('amount')
        output_required = Int('status')

    def handle(self):
        pass

# #####################################################################

class GetCurrentBalance(Service):
    """ Returns current balance of a pre-paid card.
    """
    class SimpleIO:
        input_required = Int('number')
        output_required = Int('status')
        output_optional = 'balance'

    def handle(self):
        pass

# #####################################################################

Docstrings and SimpleIO

In the sample services, observe that:

  • Documentation is added as docstrings - this is something that services, being simply Python classes, will have anyway

  • One of the services has a multi-line docstring whereas the other one's is single-line, this will be of significance later on

  • SimpleIO definitions use both string types and integers

Command line usage

To generate API specifications, command zato apispec is used. This is part of the CLI that Zato ships with.

Typically, only well-chosen services should be documented publicly, and the main two options the command has are --include and --exclude.

Both accept a comma-separated list of shell-like glob patterns that indicate which services should or should not be documented.

For instance, if the code above is saved in api.py, the command to output their API specification is:

zato apispec /path/to/server        \
    --dir /path/to/output/directory \
    --include api.*

Next, we can navigate to the directory just created and type the command below to build HTML.

cd /path/to/output/directory
make html

OpenAPI, WSDL and Sphinx

The result of the commands is as below - OpenAPI and WSDL files are in the menu column to the left.

Also, note that in the main index only the very first line of a docstring is used but upon opening a sub-page for each service its full docstring is used.

Branding and customisation

While the result is self-contained and it can be already used as-is, there is still room for more.

Given that the output is generated using Sphinx, it is possible to customise it as needed, for instance, by applying custom CSS or other branding information, such as the logo of a company exposing a given API.

All of the files used for generation of HTML are stored in config directories of each server - if the path to a server is /path/to/server then the full path to Sphinx templates is in /path/to/server/config/repo/static/sphinxdoc/apispec.

Summary

That is everything - generating static documentation is a matter of just a single command. The output can be fully customised while the resulting OpenAPI and WSDL artifacts can be given to partners to let third-parties automatically generate API clients for your Zato services.

If you need to configure Zato for Oracle DB connections and you want to ensure the highest performance possible, this is the post which goes through the process step-by-step. Read on for details.

Overview

Note that Zato 3.1+ is required. The tasks involved are:

  • Installing Zato
  • Installing an Oracle client
  • Installing cx_Oracle
  • Greenifying the Oracle client
  • Starting servers
  • Creating database connection definitions
  • Confirming the installation

Installing Zato

  • Choose your preferred operating system and follow the general installation instructions - Oracle DB connections will work the same no matter the system Zato runs on

  • Create a Zato environment with as many servers as required. Again, there are no Oracle-specific steps at this point yet.

Installing an Oracle client

  • Download an Oracle client and install it on all the systems with Zato servers

  • Add the client's installation path to LD_LIBRARY_PATH. For instance, if the client is installed to /opt/zato/instantclient_19_3, add the following to ~/.bashrc:

    export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/zato/instantclient_19_3

Installing cx_Oracle

  • cx_Oracle is a Python driver for Oracle DB, it can be installed using pip, as user zato, on each Linux instance that Zato servers use:

    $ cd /opt/zato/current
    $ ./code/bin/pip install cx_Oracle
    

Greenifying the Oracle client

  • This step is crucial for achieving the highest performance of SQL queries

  • For each Zato server installed, open its server.conf file and find stanza [greenify]

  • If there is no such stanza in server.conf, create it yourself

  • Modify the stanza to greenify the libclntsh.so library - this is a library that the Oracle client ships with. For instance, if the client's installation path is /opt/zato/instantclient_19_3, the full path to the library is /opt/zato/instantclient_19_3/libclntsh.so

  • The stanza should read as below, assuming the installation path as above:

  [greenify]
  /opt/zato/instantclient_19_3/libclntsh.so=True
  • Note that entries in this stanza are followed by =True, which is a signal to Zato that a particular library should be processed

Starting servers

  • Start Zato servers
  • For each, make sure that an entry such as below is saved in server logs

    INFO - Greenified library `/opt/zato/instantclient_19_3/libclntsh.so.19.1`
    

Creating database connection definitions

  • Go to web-admin and create a new Oracle DB connection via Connections -> Outgoing -> SQL, as below:


Create a new SQL connection

  • Update the newly created connection's password by clicking Change password

  • Click Ping to confirm the connectivity to the remote server

  • This concludes the process, the connection is ready for use in Zato services now

This article introduces features built into Zato that let one take advantage of publish/subscribe topics and message queues in communication between Zato services, API clients and backend systems.

Overview

Let's start by recalling the basic means through which services can invoke each other.

  • Using self.invoke will invoke another service directly, in a blocking manner, with the calling service waiting for a response from the target one.
  self.invoke('my.service', 'my.request')
  • With self.invoke_async, the calling service will invoke another one in background, without waiting for a response from the one being called.
  self.invoke_async('my.service', 'my.request')

There are other variants too, like async patterns, and all of them work great but what they all have in common is that the entire communication between services takes place in RAM. In other words, as long as Zato servers are up and running, services will be able to communicate.

What happens, though, when a service invokes another one and the server the other one is running on is abruptly stopped? The answer is that, without publish/subscribe, a given request will be lost irrevocably - after all, it was in RAM only so there is no way for it to be available across server restarts.

Sometimes this is desirable and sometimes it is not - publish/subscribe topics and message queues focus on scenarios of the latter kind. Let's discuss publish/subscribe, then.

Introducing publish/subscribe


In its essence, publish/subscribe is about building message-based workflows and processes revolving around asynchronous communication.

Publishers send messages to topics, subscribers have queues for data from topics and Zato ensures that messages will be delivered to subscribers even if servers are not running or if subscribers are not currently available.

Publish/subscribe and Zato services

Publish/subscribe, as it pertains to Zato services, is an extension of the idea of message topics and queues. In this case, it is either internal or user-defined services that topics and queues are used by. Whereas previously, publishers and subscribers were external applications, here both of these roles are fulfilled by services running in Zato servers.

In the diagram above, an external API client invokes a channel service, for instance a REST one. Now, instead of using self.invoke or self.invoke_async the channel service will use self.pubsub.publish to publish the message received to a backend service which will in turn deliver it to external, backend systems.

The nice part of it is that, given how Zato publish/subscribe works, even if all servers are stopped and even if some of the recipients are not available, the message will be delivered eventually. That is the cornerstone of the design - if everything works smoothly, the message will be delivered immediately, but if anything goes wrong along the way, the message is retained and attempts are made periodically to deliver it to its destination.

Python code

As usual in Zato, the Python code needed is straightforward. Below, one service publishes a message to another - the programmer does not need to think about the inner details of publish/subscribe, about locations of servers, re-deliveries or guaranteed delivery. Merely using self.publish.publish suffices for everything to work out of the box.

# -*- coding: utf-8 -*-

from __future__ import absolute_import, division, print_function, unicode_literals

# Zato
from zato.server.service import Service

# ################################################################################

class MyService(Service):
    name = 'pub.sub.source.1'

    def handle(self):

        # What service to publish the message to
        target = 'pub.sub.target.1'

        # Data to invoke the service with, here, we are just taking as-is
        # what we were given on input.
        data = self.request.raw_request

        # An optional correlation ID to assign to the published message,
        # if givenm it can be anything as long as it is unique.
        cid = self.cid

        self.pubsub.publish(target, data=data, cid=cid)

        # Return the correlation ID to our caller
        self.response.payload = cid

# ################################################################################

class PubSubTarget(Service):
    name = 'pub.sub.target.1'

    def handle(self):

        # This is how the incoming message can be accessed
        msg = self.request.raw_request

        # Now, the message can be processed accordingly
        # The actual code is skipped here - it will depend
        # on one's particular needs.

# ################################################################################

Asynchronous message flows

The kind of message flows that publish/subscribe topics promote are called asynchronous because, seeing as in a general case it is not possible to guarantee that communication will be instantaneous, the initial callers (e.g. API clients) and possibly other participants too should only submit their requests without expectations that responses, if any, will appear immediately.

Consider a simple case of topping up a pay-as-you go mobile phone. Such a process will invariably require participation from at least several backend systems, all of which can be coordinated by Zato.

Let's say that the initial caller, the API client initiating the process, is a cell phone itself, sending a text message with a top-up code from a gift card.

Clearly, there is no need for the phone itself to actively wait for the response. With several backend systems involved, it may take anything between seconds to minutes before the card is actually recharged and there is no need to keep all of the systems involved, including the cell phone, occupied.

At the same time, in case some of the backend systems are down and the initial request is lost, we cannot expect that the end user will keep purchasing more cards - we need some kind of a guaranteed delivery mechanism, which is precisely where Zato topics are useful with their ability to retain messages in case immediate delivery is not possible.

With topics, if a response is needed, instead of waiting in a synchronous manner, API callers can be given a correlation ID (CID) on output when they submit a request. A CID is just a random piece of string, uniquely identifying the request.

In the Python code example, self.cid is used for the CID. It is convenient to use it because it already available for each service and Zato knows how to use it in other parts of the platform - for instance, if the request is via HTTP (REST or SOAP), the correlation ID will be saved in Apache-style HTTP access logs. This facilitates answering of typical support questions, such as 'What happened to this or that message, when was it processed or when was the response produced?'

We have a CID but why is it useful? It is because it may be used an ID to key messages of two kinds:

  • API callers may save it and then be notified later on by Zato-based services that such and such request, one with a particular CID, has been already processed

  • API callers may save it and then periodically query Zato-based services if a given request is already processed

Which style to use ultimately depends on the overall business and technical processes that Zato publish/subscribe and services support - sometimes it is more desirable to receive notifications yet sometimes it is not possible at all, e.g. if the recipients are rarely accessible, for instance, if they join networks irregularly.

Web-admin GUI

In parting words, it needs to be mentioned that a very convenient aspect of Zato services' being part of the bigger publish/subscribe mechanism is that web-admin GUI treats them just like any other endpoint and we can browse their topics, inspect last messages published, consult web-admin to check how many messages were published, or carry out other tasks that it is capable of, like in the screenshots below: