Tutorial - part 1/2

Introduction

Note

Please read the high-level overview if you have not done it already to for an introduction of how Zato and Python help with complex and advanced systems integrations.

This Zato programming tutorial will guide you through the process of creating a real-world API service that, with Python, will integrate three applications using REST and AMQP. The result will be a solution ready to use in production.

But first, here are some real-world examples of what Zato can be used for:

  • A platform for processing payments from consumer devices
  • A system for a telecommunication operator integrating CRM, ERP, Billing and other systems as well as applications of the operator’s external partners
  • A data science system for processing of information related to securities transactions (FIX)
  • A platform for public administration systems, helping achieve healthcare data interoperability through the integration of independent data sources, databases and health information exchanges (HIE)
  • A global IoT platform integrating medical devices
  • A platform to process events produced by early warning systems
  • Backend e-commerce systems managing multiple suppliers, marketplaces and process flows
  • B2B platforms to accept and process multi-channel orders in cooperation with backend ERP and CRM systems
  • Platforms integrating real-estate applications, collecting data from independent data sources to present unified APIs to internal and external applications
  • A system for the management of hardware resources of an enterprise cloud provider
  • Online auction sites
  • E-learning platforms

What they all have in common is the level of complexity involved in integration processes. It is typical for such environments to have at least several and sometimes several dozen or several hundred independent applications exchanging messages, each possibly using a different technology and data format.

Note that Zato is a general-purpose platform and the samples span various distinct industries and domains. The platform can be used in all kinds of scenarios that require integrations of distributed systems.

API services

If Zato is a platform for API services then what is a service?

In essence, a service is a piece of server-side functionality carrying out a business action useful and interesting to applications that would like to use it. This is a high-level definition and this is the correct way to think about it.

It all stems from the fact that you should think about the systems and applications comprising your architecture in terms of consumers and providers of services. Each part of the architecture offers services to other parts and, conversely, each part consumes services offered by other elements of the architecture. Taken together, services form higher-level processes fulfilling business needs set before the IT architecture.

Zato is a multi-protocol platform and services are often not tied to any specific protocol. It means that it is possible to design services that can be invoked through REST but they can also listen for data from AMQP, IBM MQ queues or SQL databases. They can also accept HL7 MLLP, SOAP, WebSocket, SFTP, FTP, e-mail, JSON-RPC and ZeroMQ-based messages.

Naturally, REST is ubiquitous and usually this is the way that most APIs are exposed through but there are other ways too and in various scenarios other means of communication are employed.

Zato ships with connectors and adapters for REST, AWS S3, AMQP, Kafka, MongoDB, Redis, HL7, Odoo, SAP, IBM MQ, SQL, SOAP, FTP, SFTP, LDAP, Cassandra, Dropbox, Twilio, IMAP, SMTP, ElasticSearch, Solr, Swift, Slack, Telegram, WebSockets and ZeroMQ. Because it is written in Python, you have access to many third-party libraries which provide connectivity to other types of systems.

Because API platforms often need dashboards, it is also possible to use Django templates with Zato to output user interfaces.

Built-in security options include API keys, Basic Auth, JWT, NTLM, OAuth, RBAC, SSL/TLS, Vault, WS-Security and XPath. It is always possible to secure services using other, non-built in, means.

In terms of its implementation, an individual Zato service is a Python class implementing a specific method called self.handle. The service receives input, processes it according to its business requirements, which may involve communicating with other systems, applications or services, and then some output is produced. Note that both input and output are optional, e.g. a background service transferring files between applications will usually have neither whereas a typical CRUD service will have both.

Because a service is merely a Python class, it means that each one consumes very little resources and it is possible to deploy hundreds or thousands of services on a single Zato server. And because Zato can use multiple CPUs and multiple Linux instances, it scales without limits both horizontally and vertically.

Services accept their input through channels - a channel tells Zato that it should make a particular service available to the outside world using such and such protocol, data format and security definition. For instance, a service can be mounted on independent REST channels, sometimes using API keys and sometimes using Basic Auth. Additionally, each channel type has its own specific pieces of configuration, such as caching, timeouts or other options.

Services can invoke other Zato services too - this is just a regular Python method call, within the same Python process. It means that it is very efficient to invoke them - it is simply like invoking another Python method.

Services are hot-deployed to Zato servers without server restarts and a service may be made available to its consumers immediatly after deployment.

There are plugins for Visual Studio Code and PyCharm that automatically deploy your service each time you save it in your IDE. Other code editors or IDEs can be used too.

During development, usually the built-in web-admin dashboard is used to create and manage channels or other Zato objects. As soon as a solution is ready for DevOps automation, configuration of a solution can be deployed automatically from command line or directly from a git clone which makes it easy to use Zato with tools such as Terraform, Ansible or Puppet.

What will the tutorial achieve, exactly?

After completing the tutorial, we will have:

  • A complete integration environment
  • An API service offered via REST and JSON
  • The service will invoke two REST endpoints to collect data
  • The service will send notifications to an AMQP broker

Message flow

../_images/service-complete.png

We will be implementing an API integration process typical to banks and other financial institutions.

  • A client application wishes to learn details about a customer given the person’s ID
  • Customer data is stored in a CRM
  • Payments history is stored in a different application
  • For certain customer types, there is a business requirement that a fraud detection system be notified of any operations regarding such customers and we send notifications to the system accordingly

The Client App is a building block that we will not be developing in the tutorial - this is where Django, React, Vue, Flutter, ASP.NET and other frameworks can be used in actual projects.

On the other hand, remember that other backend systems can invoke the service too - this is crucial, the same one can be made available to many applications, each with its own access channel, even if in the tutorial we will assume there is only one API client.

Installing Zato

The most straightforward way to get started with Zato is installing it with Docker or Vagrant. Not only will this install Zato but the installer will automatically set up a working quickstart cluster for you too. You can use this installation method everywhere, on Windows, Mac or Linux.

Otherwise, you can install a Zato package for your selected Linux distribution and configure a quickstart cluster yourself - this will take about 2 additional minutes.

You can author Zato services with any code editor and you can also install a plugin for your IDE to auto-deploy your services on file save.

Refer to the chapters below, depending on your chosen installation method and IDE.

If you installed Zato under Docker or Vagrant

After installing Zato using Docker or Vagrant you have a container / VM with a quickstart cluster and a few TCP ports mapped to your host and one port unmapped.

../_images/quickstart-ports-docker-vm.png
Port Host-visible Purpose Notes
22022 Yes SSH For SSH connections to the Linux system the cluster runs in, just in case it is needed
8183 Yes Dashboard Web-admin dashboard used to manage Zato environments. Your browser connects to it and the dashboard itself is an API client of Zato services so it invokes servers through the load-balancer.
11223 Yes Load-balancer API clients invoke services through the load-balancer and each server always belongs to a cluster even if there is only one server in it
17010 Zato server The server is only accessible through a load-balancer which is why its TCP port is not mapped to the host directly

Before we use the environment, we will need passwords to log in to it.

Run the command below to extract them from the container or VM. Note that if you are using Docker the passwords will be regenerated each time you start a new container.

# Docker
$ docker exec zato /opt/zato/get-credentials

# Vagrant
$ cd ~/directory/with/zato/Vagrantfile
$ vagrant ssh -c 'cat  /opt/zato/get-credentials'

The result will be the same in either case:

$

SSH: oaWMFmbt5Faq6
IDE: LWdasXlqVrO8Gur00Y3ncGjmHWNLZvr
Dashboard: YUm7ufehorVEw

$

You can now go to the Dashboard at https://localhost:8183 and log in with user ‘admin’ and password printed to screen above.

We can also check if the server is running by issuing the curl command below - it will invoke a built-in service to confirm that the server started.

$ curl localhost:11223/zato/ping ; echo
{"pong":"zato","zato_env":{"result":"ZATO_OK","cid":"a5060e06b675cb20017a357d","details":""}
$

The next section details how to install Zato from a Linux package and how to create a quickstart cluster manually. You can scroll down further for information how to set up your IDE or editor now.

If you installed Zato from a Linux package

After installing Zato from a .deb or .rpm, a new user called ‘zato’ was created and all of the platform’s binaries are in /opt/zato.

To create a quickstart environment, you need to install Redis first, e.g.

$ sudo apt install redis-server

From now on, with the Zato package and Redis installed, all the commands can be executed as user zato - root powers are never needed to manage Zato environments.

Let’s create a new environment in the ~/env/qs-1 directory.

$ sudo su - zato
$ mkdir -p ~/env/qs-1
$ zato quickstart create ~/env/qs1 sqlite localhost 6379
[1/8] Certificate authority created
[2/8] ODB schema created
[3/8] ODB initial data created
[4/8] server1 created
[5/8] Load-balancer created
Superuser created successfully.
[6/8] Dashboard created
[7/8] Scheduler created
[8/8] Management scripts created
Quickstart cluster quickstart-904765 created
Dashboard user:[admin], password:[F7qCOiabas5ToQ7EWupLrHOn9iVHzyBv]
Visit https://zato.io/support for more information and support options
$

Observe the highlighted line with credentials to the Dashboard - the auto-generated password will never be shown again. In case you need to change it, run this command: zato update password ~/env/qs-1/web-admin admin.

Let’s start all the components now - note the ‘–fg’ flag when starting the server, this will start it in foreground, thanks to which we will be able to stop it with Ctrl/Cmd-C.

$ zato start ~/env/qs-1/load-balancer
OK
$

$ zato start ~/env/qs-1/web-admin
OK
$

$ zato start ~/env/qs-1/scheduler
OK
$

$ zato start ~/env/qs-1/server1 --fg
INFO - Starting Zato 3.2+rev.da807efbc-py3.6.9
INFO - Listening at: http://0.0.0.0:17010 (19475)
[...]
../_images/quickstart-ports.png

Visit http://localhost:8183, enter the credentials printed out above and browse around to get a feeling for how the Dashboard looks like.

In another terminal window, confirm that the environment is ready to accept API requests.

$ curl localhost:11223/zato/ping ; echo
{"pong":"zato","zato_env":{"result":"ZATO_OK","cid":"2fd7536d956b17fcbed24122","details":""}
$

The next step is setting up your IDE or code editor.

If you have Visual Studio Code or PyCharm

To install an IDE plugin you will first need to have credentials for the IDE user to connect to Zato with.

  • Username: ide_publisher
  • Password: Auto-generated, read below how to get it

If you are under Docker or Vagrant, run the command below to get the ide_publisher’s password (it will be different in each installation):

$ /opt/zato/get-credentials

SSH: oaWMFmbt5Faq6
IDE: LWdasXlqVrO8Gur00Y3ncGjmHWNLZvr
Dashboard: YUm7ufehorVEw

$

If you created the quickstart container yourself, the ide_publisher’s password was auto-generated so it is not possible to read it and you can reset it with the command below. This will also work if you log in to the container / VM directly and execute it from command line.

$ zato set-ide-password ~/env/qs-1/server1
aSiieQXgo6M4OS1Qv83cpHjhGR0emXN8
$

Now, you can visit installation instructions for the selected IDE.

Introducing hot-deployment

Hot-deployment is a key concept in Zato. The term means the process of transferring your service to a cluster. It is considered hot because afterwards it does not require server restarts, i.e. you hot-deploy a service and it is immediately available on all the servers.

If there is more than one server in the cluster, it suffices to hot-deploy the service to only one of them and it will synchronise with other nodes in the cluster.

There are a few ways to hot-deploy services. The first two will be used in the tutorial but we will describe each to let you understand what the options are and when to use them.

  • From your IDE

    Commonly used during development - once you install a plugin for the IDE, each time you press Ctrl/Cmd-S to save a service on disk, it will be auto-deployed to your cluster and made available for immediate use.

  • Command line

    This is used for deployment automation or if you have an IDE or editor without a Zato plugin.

    Each Zato server monitors a specific directory, called a hot-deploy directory, and each time Python files with your services are saved there, that server will pick them up and hot-deploy throughout the cluster.

    In the quickstart cluster from this tutorial, the directory is ~/env/qs-1/server1/pickup/incoming/services.

    During development, you can save your files with Zato services directly in this directory and then, when you press Ctrl/Cmd-S, the file will be deployed to the cluster. You can also clone your git repository directly into this directory.

    Another way to use it during development is to make it point to a git clone residing in another directory and again, each time you save a file its contents is sent to all the servers.

    This method is used for automation too - simply use cp to copy files into the directory and all the services from these files will be deployed.

  • Dashboard

    When you log in to Dashboard and navigate to Services, you will note a button called “Upload services”. This will let you deploy local files to a remote server. This is useful when there is no direct connection to the server, e.g. no way to ssh into it.

  • Local config file

    This method is usually used for automated deployments only - it lets you point a starting server to files from the file system that it should deploy.

    The difference between it and a hot-deploy directory is that the latter requires for the server to be already running whereas this one tells a server what it should do while it is still starting up.

    This option is most often employed when building one’s own Docker images or using Terraform, Packer and similar tools.

  • Remote file transfer

    This automation method uses file transfer to let servers listen for changes in directories of remote servers.

    For instance, you can have a central git clone of a repository for multiple environments and Zato servers will connect to it via SFTP, download any new or changed ones and deploy them locally.

In terms of the end result, there is no difference between the methods, they achieve exactly the same result.

This is actually a good example of the way Zato itself is designed around reusable services - all these deployment methods, all these channels, ultimately lead to the same services that deploy your code and it is only the manner in which they are accessed that differs.

Hot-deploying your first service

We can now create the first service and hot-deploy it. Create a new file called api.py with the contents below. This is a basis of the service that we will fill in with details later on.

# -*- coding: utf-8 -*-
# zato: ide-deploy=True

from zato.server.service import Service

class GetUserDetails(Service):
    """ Returns details of a user by the person's ID.
    """
    name = 'api.user.get-details'

    def handle(self):

        # For now, return static data only
        self.response.payload = {
            'user_name': 'John Doe',
            'user_type': 'SRT'
        }

If you configured a plugin for PyCharm or Visual Studio Code, note the hlighlighted line - this is a special marker which lets the plugin know that saving this file should result in the IDE’s deploying it to your cluster.

Without a plugin, you need to save the file in the server’s ~/env/qs-1/server1/pickup/incoming/services directory. If you use Docker and Vagrant but you do not have the plugin, you first need to map this directory to one in the host system and save the file in the latter.

And if you want to deploy it to a Zato server from your browser, log in to the Dashboard, go to Services and click Upload package.

Dashboard’s address is http://localhost:8183, username is admin and if you need to reset the password, this is the command:

$ zato update password ~/env/qs-1/web-admin admin

Having deployed the code, we can confirm in the Dashboard that the service is there.

Once you logged in to the Dashboard, navigate to Services -> Enter “get-details” in the search box -> Show services -> Click View and this will display basic information about the deployed service. You can click Source code to confirm that this is the same service.

We have a service so now we can create a REST channel for it.

Creating your first channel

We want to invoke our API service using REST but we also want to make sure that access to it is secured so we will first create a security definition for our API client.

In Dashboard, go to Security -> Basic Auth -> Click “Create a new definition” and enter:

  • Name: API Credentials
  • Username: api
  • Domain: API

Clicking OK will create the definition with its user’s password automatically set to a random uuid4 so we need to reset it by clicking “Change password” and providing a new one - it is up to you to decide what it should be.

Now, we can create a REST channel by going to Connections -> Channels -> REST, as below:

../_images/rest-menu.png

Click Create a new REST channel link:

../_images/rest-create-link.png

Fill out the form as here, the fields to provide values for are:

  • Name
  • URL path
  • Data format
  • Service
  • Security definition
../_images/rest-create-form.png

Clicking OK will create the channel and we will be able to invoke the API service now.

Invoking your first service

We are going to use curl to invoke the service - we will access it through the load-balancer’s port of 11223, as below. Note that you need to enter the API client’s password too.

$ curl http://api:<password-here>@localhost:11223/api/v1/user ; echo
{"user_name":"John Doe","user_type":"SRT"}
$

Everything works as expected - you have just created and invoked your first API service! Now, try to see what happens if you provide an invalid password or a URL path - your requests will not be allowed.

This concludes the first part of the tutorial and the next one will see the service integrate with external systems to transform and enrich their replies before producing the final response to the API client.

But first, there is one observation to be made - the quickstart cluster that you created is a real, fully functional environment. If you were to create it from scratch, by adding each component individually, Dashboard, server, load-balancer and scheduler, the outcome would be precisely the same.

In other words, quickstart clusters are a convenient method for creation of new environments and they can be very well used not only for development but for testing and production too.

Now, we are ready to go to the second part of the tutorial.