Tutorial - part 2/2

Be sure to complete part 1 of the tutorial first. Some ideas previously explained there are not repeated here.

Invoking other systems

To summarize it, in the previous part we set up a working Zato cluster, deployed a service and created a REST API channel through which the service was invoked.

In this part, we will make the service actually obtain data from remote systems and process it in accordance with its business requirements. Initial chapters of this part of the tutorial will mostly use the Dashboard to configure external connections and we will return to Python afterwards.

To invoke other systems, applications and APIs, Zato services use outgoing connections which is the concept we will cover now.

Outgoing connections

Outgoing connections are the natural counterpart to channels. Whereas channels allow one to make Zato services become available to external API clients, with outgoing connections (outconns for short) it is Zato services that invoke endpoints of external systems.

Outconns are typically invoked using attributes from self.out, e.g. self.out.rest, self.out.amqp, self.out.sap and so on, maintaining a connection pool internally when needed so that services can just focus on the invocation part.

Outgoing connections, like channels or other Zato elements, let you insulate the business logic of services from their configuration. As we will see later, a service only refers to abstract names like "CRM" when it wants to access some external resource, without a need for it to know where the CRM actually is, under what address and secured with what credentials.

Separating logic from configuration lets you deploy the same unchanged code to multiple environments. It also means that it is easy to migrate to new or modified environments. For instance, if today your service connects to a REST API with a Basic Auth security definition but tomorrow the API will be secured with TLS private keys, it will be only a matter of an update to configuration and the service will continue to work uninterrupted, without any downtime.

Speaking of configuration, throughout the tutorial we mostly use the Dashboard to manage configuration but it can be exported to YAML or JSON and imported in other environments too, we will talk about it later.

In addition to outconns, it is also possible to install libraries from PyPI and invoke remote systems using client libraries for connection types other than what Zato has built-in.

For the purposes of the tutorial, all the REST and AMQP endpoints are already prepared for you so that you do not need to set up anything and we can start by creating the outconns now.

REST outgoing connections

In Dashboard at http://localhost:8183, go to Connections → Outgoing → REST and click Create a new REST outgoing connection.

A form will pop up, fill it out per the table below. Note the default HEAD ping method, we are going to make use of it in the next section.

We need a REST outgoing connection to the Payments system too, as in the following table.

Pinging REST outgoing connections

Having created REST outconns, we can check if they have connectivity to the systems they point to by pinging them - there is a Ping link for each outconn.

Click it and confirm that the response is similar to the one below - as long as it is in green, the connection works fine.

The connection is pinged not from your localhost but from one of the servers in your cluster - in this way you can confirm that it truly is your servers, rather than your local system, that has access to a remote endpoint.

AMQP outgoing connections

AMQP connections are created similarly to the REST ones except that instead of going directly to outgoing connections, we first visit Connections → Definitions → AMQP in the Dashboard.

Connection definitions are reusable configuration objects that are employed if a given technology or protocol can be used in both channels and outgoing connections which is the case with AMQP. The tutorial only sends messages to AMQP but in another project, you may have both AMQP channels and outconns pointing to the same broker and one definition encompasses configuration for both.

Having a single definition with credentials for both types makes it convenient to update the common parts of the configuration in one place only, e.g. after you change a username or host in an AMQP definition, all channels and outgoing connections using this definition will auto-reconfigure and reconnect as needed.

Thus, go to Connections → Definitions → AMQP and create a new definition as in this table.

This created a new AMQP connection definition and we need to set the user's password. Click Change password and enter PJs4TEeui118A - note that the password changes periodically and it may be different if you visit the tutorial in some time.

At this point, we have a definition but by itself, it will not exchange messages with an AMQP broker, it is only channels or outconns that to do it using configuration and credentials from their parent definitions.

Hence, create a new AMQP outconn at Connections → Outgoing → AMQP. Except for the two specific values below you can leave the rest unchanged with blank or default values.

Pinging AMQP with publications

We can ping a remote AMQP broker by publishing a message to it. Go to Connections → Outgoing → AMQP and click Publish in the table that will display.

Now, send a test message using the configuration below.

There will be a response on a green background confirming that the message was published successfully.

All the outgoing connections are created so we can now shift our attention back to the Python service.

Back to the service

The logic of our service shall be:

• Accept user_name on input
• Invoke CRM to get basic user data, including account_number and user_type
• Invoke Payments to get account details by account_number
• If user_type matches configuration, notify Fraud detection

In a bigger integration project, an individual user would have more than one bank account but here we keep it simple and assume that each user has one bank account.

Here is the full Python implementation of the logic above.

# -*- coding: utf-8 -*-
# zato: ide-deploy=True

# Zato
from zato.server.service import Service

# ##############################################################################

class GetUserDetails(Service):
    """ Returns details of a user by the person's ID.
    """
    name = 'api.user.get-details'

    def handle(self):

        # For later use
        user_name = self.request.payload['user_name']

        # Get data from CRM ..
        crm_data = self.invoke_crm(user_name)

        # .. extract the CRM information we are interested in ..
        user_type = crm_data['UserType']
        account_no = crm_data['AccountNumber']

        # .. get data from Payments ..
        payments_data = self.invoke_payments(user_name, account_no)

        # .. extract the CRM data we are interested in ..
        account_balance = payments_data['ACC_BALANCE']

        # .. optionally, notify the fraud detection system ..
        if self.should_notify_fraud_detection(user_type):
            self.notify_fraud_detection(user_name, account_no)

        # .. now, produce the response for our caller.
        self.response.payload = {
          'user_name': user_name,
          'user_type': user_type,
          'account_no': account_no,
          'account_balance': account_balance,
      }

# ##############################################################################

    def invoke_crm(self, user_name):

        # Log what we are about to do
        self.logger.info('Invoking CRM; u=%s', user_name)

        # Obtain a connection to CRM ..
        conn = self.out.rest['CRM'].conn

        # .. produce a request for CRM ..
        request = {
            'UserName': user_name,
        }

        # .. invoke the CRM ..
        crm_response = conn.get(self.cid, request)

        # .. return data received from CRM.
        return crm_response.data

# ##############################################################################

    def invoke_payments(self, user_name, account_no):

        # Log what we are about to do
        self.logger.info('Invoking Payments; u=%s, a=%s', user_name, account_no)

        # Obtain a connection to Payments ..
        conn = self.out.rest['Payments'].conn

        # .. prepare a request for Payments ..
        request = {
            'ACC_NUM': account_no,
        }

        # .. prepare query string parameters ..
        params = {'USER': user_name}

        # .. invoke Payments ..
        response = conn.post(self.cid, params=params)

        # .. return data received from Payments.
        return response.data

# ##############################################################################

    def notify_fraud_detection(self, user_name, account_no):

        # Log what we are about to do
        self.logger.info('Notifying Fraud detection; u=%s', user_name)

        # Data to send to the Fraud detection system
        data = 'User `{}` accessed `{}`'.format(user_name, account_no)

        # AMQP configuration
        outconn = 'Fraud Detection Connection'
        exchange = '/tutorial'
        routing_key = 'api'

        # Send the message to Fraud Detection
        self.out.amqp.send(data, outconn, exchange, routing_key)

# ##############################################################################

    def should_notify_fraud_detection(self, user_type):
      config = self.server.user_config.get('tutorial')
      if config:
          return config.notify_fraud.get(user_type)

# ##############################################################################

We can immediately observe that:

• This is high-level code - implementation-wise, we operate on the level of Python dicts and a few methods

• The service is to a great part independent of the underlying systems it needs to integrate; it does not concern itself with details of any of the protocols that are used in this integration effort

• We use simple methods, like .get, .post or .send to connect to external systems and it is the job of the Zato platform to translate it into actual invocations, e.g. the service only sends data to 'CRM', 'Payments or 'Fraud Detection Connection' but it does not need to know which REST endpoint or an AMQP broker is meant by that exactly

• We check whether to send notifications to the fraud detection system using self.server.user_config - this points to a local config file that we will create in just a moment. But, because this file does not exist yet, the should_notify_fraud_detection method checks whether it can make use of it.

Invoking the service

Let's use curl to access the service:

# Make sure to use here the password set in the first part of the tutorial
c:\>curl -XPOST http://api:<password>@localhost:17010/api/v1/user \
      -d '{"user_name": "my.user"}'

# Here is our response
{"user_name":"my.user","user_type":"RGV","account_no":"123456","account_balance":"357.9"}
c:\>

In server logs (also found in ~/AppData/Local/Zato3.2/env/qs-1/server1/logs/server.log), we can now observe:

INFO - Invoking CRM; u=my.user
INFO - Invoking Payments; u=my.user, a=123456

We are almost done now - the only remaining part is ensuring that the Fraud detection system is notified when needed. For that, we need to discuss our runtime configuration options.

Runtime configuration

Each project needs to keep its runtime configuration in some location. What is stored in such configuration are project-specific details like business rules or conditions.

Broadly, there are two kinds of locations where it can be kept. Both kinds are a good choice and it is mostly an architect's choice of what to use.

• Configuration files
• Databases, such as Redis, MongoDB, Vault, SQL or similar ones

It is worth to say that in Zato configuration files are reloaded automatically and cached in RAM each time they are updated. This means that you can change their contents without server restarts and without a need to redeploy your services.

To keep it simple, so as not to require the installation of additional components such as Redis, we are going to use a config file.

Save the below to ~/AppData/Local/Zato3.2/env/qs-1/server1/config/repo/user-conf/tutorial.conf:

[notify_fraud]
RGV=True

Such a file will be picked up by Zato and its contents will be available to a service through self.server.user_config, e.g. to access the value of RGV, you can use self.server.user_config.tutorial.notify_fraud.RGV.

There are several significant advantage of using config files:

• Clarity of intent - it is easy to read an .ini file

• Performance - the contents of such files in automatically synced to RAM each time a file changes on disk which means that there are no runtime lookups in a remote database and no disk reads

• Each file is converted to a dict-like structure which, at the same time, can be accessed via dot syntax, i.e. all dict methods will continue to work and you can check whether keys exist in a file, for instance

• Ability to store configuration in a code repository - simply push a file to git repo and then pull it in the server's user-conf directory

• Config files are regular text files which means that they can be generated from other tools, e.g. if you use a UML tool for your modeling, it is always possible to generate such a config file from a given tool's native format

If we now invoke the service again, we will notice a new entry in server logs because this time around we are sending an AMQP message too (note the highlighted line):

INFO - Invoking CRM; u=my.user
INFO - Invoking Payments; u=my.user, a=123456
INFO - Notifying Fraud detection; u=my.user

As far as the implementation goes, we are done, this is everything. What we have now is a fully working integration platform and a service integrating API clients with three backend systems.

More capabilities

We have only covered the tip of the iceberg in terms of Zato offers so let's quickly list all the various features that are available for building API and server-side systems.

• Message broker with publish/subscribe topics and message queues
• File transfer
• Task scheduler
• REST and Python Single Sign-On
• Statistics
• Channels: AMQP, HL7, IBM MQ, JSON-RPC, REST, SOAP, WebSockets and ZeroMQ
• Outgoing connections: AMQP, FTP, HL7, IBM MQ, LDAP, Redis, MongoDB, Odoo, REST, SAP RFC, SFTP, SOAP, SQL, WebSockets and ZeroMQ
• More connection types: AWS S3, Dropbox, Built-in cache, Memcached, ElasticSearch, Solr, Swift, Cassandra, Slack, Telegram, IMAP, SMTP and Twilio
• Security: API keys, HTTP Basic Auth, JWT, NTLM, OAuth, RBAC, SSL/TLS, Vault, WS-Security and XPath
• CLI and API offered as services (e.g. Dashboard itself is an API client of the platform's own public services)
• Numerous smaller tools and utilities

Let's also have a quick look at two features in particular - deployment automation and API testing.

DevOps automation

Throughout the tutorial, we have been only using the Dashboard to configure Zato objects, such as channels or outgoing connections. This is good but when it comes to automated deployment, we would like to have a way to export our configuration and import it in another environment. This is just what we will do now.

Among other command-line options is zato enmasse - this command lets you export definitions of Zato objects, merge multiple ones and import them in other environments. The data format it uses is either YAML or JSON, depending on one's preferences.

The typical workflow with enmasse is to progressively keep exporting definitions from a development environment and store them in a code repository, exporting them to other environments when the time comes. Because they are simple human-readable files, it is easy to diff them or replace their contents during deployment.

In practice, enmasse is used as here:

c:\>zato enmasse ~/AppData/Local/Zato3.2/env/qs-1/server1 --export-odb
ODB objects read
ODB objects merged in
Data exported to /opt/zato/zato-export-2022-01-29T15_27_35_609729.yml
c:\>

The resulting file will contain definitions of all the objects found in the cluster's database. For instance, we can find our AMQP outgoing connection among them:

outconn_amqp:

  - def_name: 'Fraud Detection Definition'
    name: 'Fraud Detection Connection'
    priority: 5

After pre-processing, such a file can be stored in git. Moreover, to reduce any chance for conflicts, multiple developers can each have his or her own file with configuration and enmasse will combine them when it is time to import the objects as a whole.

As to how importing works, it is akin to the previous call. Note the usage of the --replace-odb-objects flag - this tells enmasse to update already existing objects in-place, otherwise it would refuse to continue lest it was not actually your intention to replace what is already stored in the cluster's Operational Database (ODB).

c:\>zato enmasse ~/AppData/Local/Zato3.2/env/qs-1/server1 --import \
  --input ./zato-export-2022-01-29T15_27_35_609729.yml \
  --replace-odb-objects
Invoking zato.outgoing.amqp.edit for outconn_amqp
Updated object `Fraud Detection Connection`
c:\>

API testing

We have an implementation, configuration and automated deployment so now we can add API tests.

Zato has a command-line tool called apitest which lets you write tests in pure English and we can use it to test the integration process developed in this tutorial:

Let's copy the test below to make it easier to analyze it:

Feature: Zato Tutorial

Scenario: *** Call api.user.get-details ***

    Given address "http://localhost:17010"
    Given Basic Auth "api" "<password>"
    Given URL path "/api/v1/user"
    Given format "JSON"
    Given HTTP method "GET"
    Given request is "{}"
    Given path "/user_name" in request is "my.user"

    When the URL is invoked

    Then path "/user_name" is "my.user"
    And path "/user_type" is "RGV"
    And path "/account_no" is "123456"
    And path "/account_balance" is "357.9"
    And status is "200"
    And header "X-Zato-CID" is not empty

• All API tests are written in English, without a need for any programming - this makes it easy for everyone to participate in their creation

• Tests are broken out into features and scenarios. E.g. a feature may be "User Account Management" with individual scenarios testing CRUD and other parts of the functionality.

• Scenarios may relay information and context across tests, e.g. it is possible to have a setup scenario preparing data to test with, followed by tests and ending with a tear down phase that cleans up test data. This is just like with unit-tests except that here we test entire APIs using live environments

• It is possible to form a chain of tests invocations, e.g. one test may get user data, extract what is needed and pass it on to another test that needs this information

• Tests configuration can be kept in external files or environment variables - it means that the same tests can be used with different input, output or different environments

• Because apitest runs from the command line, it is easy to plug it into any kind of a development and deployment pipeline

This concludes the tutorial - you are now ready to explore the platform further and build your own API solutions using Zato.