Tutorial - part 1/2

• Este tutorial también está disponible en español
• Ce tutoriel est aussi disponible en français

Introduction

Please read the high-level overview if you have not done it already for an introduction of how Zato and Python help with complex and advanced systems integrations.

This tutorial will guide you through the process of creating a real-world API service that, with Python, will integrate three applications using REST and AMQP. The result will be a solution ready to use in production.

But first, here are some real-world examples of what Zato can be used for:

• A platform for processing payments from consumer devices.
• Systems for telecommunication operators integrating CRM, ERP, Charging Systems, Billing and other OSS/BSS applications internal or external to the operators.
• A data science system for processing of information related to securities transactions (FIX).
• A platform for public administration systems, helping achieve healthcare data interoperability through the integration of independent data sources, databases and health information exchanges (HIE).
• A global IoT platform integrating medical devices.
• A platform to process events produced by early warning systems.
• Backend e-commerce systems managing multiple suppliers, marketplaces and process flows.
• B2B platforms to accept and process multi-channel orders in cooperation with backend ERP and CRM systems.
• Platforms integrating real-estate applications, collecting data from independent data sources to present unified APIs to internal and external applications.
• A system for the management of hardware resources of an enterprise cloud provider.
• Online auction sites.
• E-learning platforms.

What they all have in common is the level of complexity involved in integration processes. It is typical for such environments to have at least several and sometimes several dozen or several hundred independent applications exchanging messages, each possibly using a different technology and data format.

Note that Zato is a general-purpose platform and the samples span various distinct industries and domains. The platform can be used in all kinds of scenarios that require automation and integrations of distributed systems.

API services

If Zato is a platform for API services then what is a service?

In essence, a service is a piece of server-side functionality carrying out a business action useful and interesting to applications that would like to use it. This is a high-level definition and this is the correct way to think about it.

It all stems from the fact that you should think about the systems and applications comprising your architecture in terms of consumers and providers of services. Each part of the architecture offers services to other parts and, conversely, each part consumes services offered by other elements of the architecture. Taken together, services form higher-level processes fulfilling business needs set before the IT architecture.

Zato is a multi-protocol platform and services are often not tied to any specific protocol. It means that it is possible to design services that can be invoked through REST but they can also listen for data from AMQP, IBM MQ queues or SQL databases. They can also accept HL7 MLLP, SOAP, WebSocket, SFTP, FTP, e-mail, JSON-RPC and ZeroMQ-based messages.

Naturally, REST is ubiquitous and usually, this is the way that most APIs are exposed through but there are other ways too and in various scenarios other means of communication are employed.

Zato ships with connectors and adapters for REST, AWS S3, AMQP, Kafka, MongoDB, Redis, HL7, Odoo, SAP, IBM MQ, SQL, SOAP, FTP, SFTP, LDAP, Cassandra, Dropbox, Twilio, IMAP, SMTP, ElasticSearch, Solr, Swift, Slack, Telegram, WebSockets and ZeroMQ. Because it is written in Python, you have access to many third-party libraries which provide connectivity to other types of systems.

Because API platforms often need dashboards, it is also possible to use Django templates with Zato to output user interfaces.

Built-in security options include API keys, Basic Auth, JWT, NTLM, OAuth, RBAC, SSL/TLS, Vault, WS-Security and XPath. It is always possible to secure services using other, non-built in, means.

In terms of its implementation, an individual Zato service is a Python class implementing a specific method called self.handle. The service receives input, processes it according to its business requirements, which may involve communicating with other systems, applications or services, and then some output is produced. Note that both input and output are optional, e.g. a background service transferring files between applications will usually have neither whereas a typical CRUD service will have both.

Because a service is merely a Python class, it means that each one consumes very little resources and it is possible to deploy hundreds or thousands of services on a single Zato server. And because Zato can use multiple CPUs and multiple Linux instances, it scales without limits both horizontally and vertically.

Services accept their input through channels - a channel tells Zato that it should make a particular service available to the outside world using such and such protocol, data format and security definition. For instance, a service can be mounted on independent REST channels, sometimes using API keys and sometimes using Basic Auth. Additionally, each channel type has its own specific pieces of configuration, such as caching, timeouts or other options.

Services can invoke other Zato services too - this is just a regular Python method call, within the same Python process. It means that it is very efficient to invoke them - it is simply like invoking another Python method.

Services are hot-deployed to Zato servers without server restarts and a service may be made available to its consumers immediately after deployment.

There are plugins for Visual Studio Code and PyCharm that automatically deploy your service each time you save it in your IDE. Other code editors or IDEs can be used too.

During development, usually, the built-in web-admin dashboard is used to create and manage channels or other Zato objects. As soon as a solution is ready for DevOps automation, a configuration of a solution can be deployed automatically from the command line or directly from a git clone which makes it easy to use Zato with tools such as Terraform, Nomad or Ansible.

What will the tutorial achieve, exactly?

After completing the tutorial, we will have:

• A complete integration environment.
• An API service offered via REST and JSON.
• The service will invoke two REST endpoints to collect data.
• The service will send notifications to an AMQP broker.

Message flow

We will be implementing an API integration process typical to banks and other financial institutions.

• A client application wishes to learn details about a customer given the person's ID.
• Customer data is stored in a CRM.
• Payments history is stored in a different application.
• For certain customer types, there is a business requirement that a fraud detection system be notified of any operations regarding such customers and we send notifications to the system accordingly.

The Client App is a building block that we will not be developing in the tutorial - this is where Django, React, Vue, Flutter, ASP.NET and other frameworks can be used in actual projects.

On the other hand, remember that other backend systems can invoke the service too - this is crucial, the same one can be made available to many applications, each with its own access channel, even if in the tutorial we will assume there is only one API client.

Installing Zato

The installation procedure for Mac, Linux, Docker and Vagrant is covered in the other version of the tutorial.

Under Windows, follow the steps below. Administrator rights are not required.

• Download the Zato installer by clicking here
• Unpack the .zip archive and run the "install.bat" file that you will find inside
• Click "Install" to begin the installation process which will install Zato and create a new environment for you. Click "Exit" once it finishes.

• Afterwards, click "Start Zato" either through a newly created desktop icon or through a new entry in the Start menu.

• Clicking "Start Zato" will start your environment with server logs visible in a new window - to stop the environment, simply close the window.

What did the installer install?

Installing Zato creates a quickstart cluster as well. Quickstart clusters are a convenient way of creating new, self-contained, fully functional environments comprised of one server, dashboard and a scheduler.

You can create quickstart clusters from the command line and via Docker Quickstart as well and they can be used for any purposes, from development, through testing to production.

The two main endpoints of a Windows quickstart cluster are:

Managing your credentials

As a rule, there are no default credentials anywhere in Zato, which means that each environment has different, auto-generated passwords.

To retrieve the one generated for you, click Programs → Zato 3.2 → Show configuration in the Windows Start menu. This will open a JSON file with various details of the environment, including two keys that are of interest now:

Note that you can also change the passwords at any time directly in the JSON file and they will be used when you restart the environment by closing its window and clicking "Start Zato" again.

You can now go to the Dashboard at http://localhost:8183 and log in with user 'admin' and password from the configuration file.

Where is Zato installed on disk?

Zato itself and the newly created environment is installed to your Windows user's ~\AppData\Local directory.

The exact name of this directory may differ depending on the locale set for your Windows system. If the language is English and the username is "dsuch" then the default, full path will be C:\Users\dsuch\AppData\Local\Zato3.2, as in the screenshot below.

Note that, because Zato does not require Administrator permissions, it never installs anything to "C:\Program Files" or similar locations.

Integrating with your IDE

You can author Zato services with any code editor and you can also install a plugin for your IDE to auto-deploy your services each time a Python file is saved on disk.

IDE	Notes
	Configure Visual Studio Code for work with Zato.
	Configure PyCharm for work with Zato.

Depending on the plugin's version, it may by default try to connect to localhost:11223 instead of localhost:17010 because 11223 is the default plugin port under Mac, Linux and Docker. If that be the case, make sure to configure the plugin to use localhost:17010 which is what Zato for Windows uses.

Regardless of what your IDE and plugin are, the username for the plugin to authenticate with a Zato server is always "ide_publisher".

Introducing hot-deployment

Hot-deployment is a key concept in Zato. The term means the process of transferring your service to a cluster. It is considered hot because, afterwards, it does not require server restarts, i.e. you hot-deploy a service and it is immediately available on all the servers.

If there is more than one server in the cluster, it suffices to hot-deploy the service to only one of them and it will synchronize with other nodes in the cluster.

There are a few ways to hot-deploy services. The first two will be used in the tutorial but we will describe each to let you understand what the options are and when to use them.

• From your IDE

  Commonly used during development - once you install a plugin for the IDE, each time you press Ctrl-S to save a service on disk, it will be auto-deployed to your cluster and made available for immediate use.

• Command line

  This is used for deployment automation or if you have an IDE or editor without a Zato plugin.

  Each Zato server monitors a specific directory, called a hot-deploy directory, and each time Python files with your services are saved there, that server will pick them up and hot-deploy throughout the cluster.

  In the quickstart cluster from this tutorial, the directory is ~\AppData\Local\Zato3.2\env\qs-1\server1\pickup\incoming\services.

  During development, you can save your files with Zato services directly in this directory and then, when you press Ctrl-S, the file will be deployed to the cluster. You can also clone your git repository directly into this directory.

  Another way to use it during development is to make it point to a git clone residing in another directory and again, each time you save a file its contents are sent to all the servers.

  This method is used for automation too - simply use built-in copy or robocopy Windows tools to copy files into the directory and all the services from these files will be deployed.

• Dashboard

  When you log in to the Dashboard and navigate to Services, you will note a button called "Upload services". This will let you deploy local files to a remote server. This is useful when there is no direct connection to the server, e.g. no way to ssh into it.

• Local config file

  This method is usually used for automated deployments only - it lets you point a starting server to files from the file system that it should deploy.

  The difference between it and a hot-deploy directory is that the latter requires for the server to be already running whereas this one tells a server what it should do while it is still starting up.

  This option is most often employed when building one's own Docker images or using Terraform, Packer and similar tools.

• Remote file transfer

  This automation method uses file transfer to let servers listen for changes in directories of remote servers.

  For instance, you can have a central git clone of a repository for multiple environments and Zato servers will connect to it via SFTP, download any new or changed ones and deploy them locally.

In terms of the end result, there is no difference between the methods, they achieve exactly the same result.

This is actually a good example of the way Zato itself is designed around reusable services - all these deployment methods, all these channels, ultimately lead to the same services that deploy your code and it is only the manner in which they are accessed that differs.

Hot-deploying your first service

We can now create the first service and hot-deploy it. Create a new file called api.py with the contents below. This is a basis of the service that we will fill in with details later on.

# -*- coding: utf-8 -*-
# zato: ide-deploy=True

from zato.server.service import Service

class GetUserDetails(Service):
    """ Returns details of a user by the person's ID.
    """
    name = 'api.user.get-details'

    def handle(self):

        # For now, return static data only
        self.response.payload = {
            'user_name': 'John Doe',
            'user_type': 'SRT'
        }

If you configured a plugin for PyCharm or Visual Studio Code, note the highlighted line - this is a special marker which lets the plugin know that saving this file should result in the IDE's deploying it to your cluster.

Without a plugin, you need to save the file in the server's ~/env/qs-1/server1/pickup/incoming/services directory.

If you want to deploy it to a Zato server from your browser, log in to the Dashboard, go to Services and click Upload package. Dashboard's address is localhost:8183, username is "admin" and the password is in your Start menu, under Programs → Zato 3.2 → Show configuration.

No matter how you deploy the service, there will be activity in the server's log:

Having deployed the code, we can confirm in the Dashboard that the service is there.

Once you logged in to the Dashboard, navigate to Services. Enter "get-details" in the search box, then click Show services. Click the service's name and this will display basic information about the deployed service. You can click Source code to confirm that this is the same service.

We have a service so now we can create a REST channel for it.

Creating your first channel

We want to invoke our API service using REST but we also want to make sure that access to it is secured so we will first create a security definition for our API client.

In Dashboard, go to Security → Basic Auth → Click "Create a new definition" and enter:

• Name: API Credentials
• Username: api
• Domain: API

Clicking OK will create the definition with its user's password automatically set to a random uuid4 so we need to reset it by clicking "Change password" and providing a new one - it is up to you to decide what it should be.

Now, we can create a REST channel by going to Connections → Channels → REST, as below:

Click Create a new REST channel link:

Fill out the form as here, the fields to provide values for are:

• Name
• URL path
• Data format
• Service
• Security definition

Clicking OK will create the channel and we will be able to invoke the API service now.

Invoking your first service

We are going to use curl to invoke the service - we will access it through the server's port of 17010, as below. Note that you need to enter the API client's password too.

$ curl http://api:<password-here>@localhost:17010/api/v1/user ; echo
{"user_name":"John Doe","user_type":"SRT"}
$

Everything works as expected - you have just created and invoked your first API service! Now, try to see what happens if you provide an invalid password or a URL path - your requests will not be allowed.

This concludes the first part of the tutorial and the next one will see the service integrate with external systems to transform and enrich their replies before producing the final response to the API client.

But first, there is one observation to be made - the quickstart cluster that you created is a real, fully functional environment. If you were to create it from scratch, by adding each component individually, Dashboard, server and scheduler, the outcome would be precisely the same.

In other words, quickstart clusters are a convenient method for the creation of new environments and they can be very well used not only for development but for testing and production too.

Now, we are ready to go to the second part of the tutorial.