Python API Integration Insights with Zato

SSH API Service in Python

Mon, 12 Jan 2026 03:00:00 +0000

SSH API Service in Python

2026-01-12, by Dariusz Suchojad

This is a quick guide on how to turn SSH commands into a REST API service. The use-case may be remote administration of devices or equipment that does not offer a REST interface or making sure that access to SSH commands is restricted to selected external REST-based API clients only.

Python

The first thing needed is code of the service that will connect to SSH servers. Below is a service doing just that - it receives name of the command to execute and host to run in on, translating stdout and stderr of SSH commands into response documents which Zato in turn serializes to JSON.

# -*- coding: utf-8 -*-

# stdlib
from traceback import format_exc

# Zato
from zato.server.service import Service

class SSHInvoker(Service):
    """ Accepts an SSH command to run on a remote host and returns its output to caller.
    """

    # A list of elements that we expect on input
    input = 'host', 'command'

    # A list of elements that our responses will contain
    output = 'is_ok', 'cid', '-stdout', '-stderr'

    def handle(self):

        # Local aliases
        host = self.request.input.host
        command = self.request.input.command

        # Correlation ID is always returned
        self.response.payload.cid = self.cid

        try:
            # Build the full command
            full_command = f'ssh {host} {command}'

            # Run the command and collect output
            output = self.commands.invoke(full_command)

            # Assign both stdout and stderr to response
            self.response.payload.stdout = output.stdout
            self.response.payload.stderr = output.stderr

        except Exception:
            # Catch any exception and log it
            self.logger.warn('Exception caught (%s), e:`%s', self.cid, format_exc())

            # Indicate an error
            self.response.payload.is_ok = False

        else:
            # Everything went fine
            self.response.payload.is_ok = True

Dashboard

In the Zato Dashboard, let's go ahead and create an HTTP Basic Auth definition that a remote API client will authenticate against:

Now, the SSH service can be mounted on a newly created REST channel - note the security definition used and that data format is set to JSON. We can skip all the other details such as caching or rate limiting, for illustration purposes, this is not needed.

Usage

At this point, everything is ready to use. We could make it accessible to external API clients but, for testing purposes, let's simply invoke our SSH API gateway service from the command line:

$ curl "api:password@localhost:11223/api/ssh" -d \
    '{"host":"localhost", "command":"uptime"}'
{
    "is_ok": true,
    "cid": "27406f29c66c2ab6296bc0c0",
    "stdout": " 09:45:42 up 37 min,  1 user,  load average: 0.14, 0.27, 0.18\n"}
$

Note that, at this stage, the service should be used in trusted environments only, e.g. it will run any command that it is given on input which means that in the next iteration it could be changed to only allow commands from an allow-list, rejecting anything that is not recognized.

And this completes it - the service is deployed and made accessible via a REST channel that can be invoked using JSON. Any command can be sent to any host and their output will be returned to API callers in JSON responses.

More resources

➤ Python API integration tutorials
➤ What is an integration platform?
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?
➤ Open-source iPaaS in Python

Python scheduler for API integrations

Mon, 05 Jan 2026 03:00:00 +0000

Python scheduler for API integrations

2026-01-05, by Dariusz Suchojad

Are you looking for a practical way to automate API tasks with reliable scheduling? This tutorial demonstrates how to implement task scheduling using Zato within Docker.

The guide focuses on real-world implementation rather than theory, showing you how to build scheduling solutions that work consistently in production environments.

What you'll learn in the tutorial:

How to set up a Python scheduler with Docker
Creating services that run on precise intervals (as short as 5 seconds)
Techniques for API automation, SQL database integration, and Bash script execution
DevOps practices for maintaining scheduled jobs across environments

More resources

Modern REST API Tutorial in Python

Mon, 22 Dec 2025 03:00:00 +0000

Modern REST API Tutorial in Python

2025-12-22, by Dariusz Suchojad

Great APIs don't win theoretical arguments - they just prefer to work reliably and to make developers' lives easier.

Here's a tutorial on what building production APIs is really about: creating interfaces that are practical in usage, while keeping your systems maintainable for years to come.

Sound intriguing? Read the modern REST API tutorial in Python here.

More resources

➤ Python API integration tutorials
➤ What is a Network Packet Broker? How to automate networks in Python?
➤ What is an integration platform?
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?
➤ Open-source iPaaS in Python

Microsoft Dataverse with Python and Zato Services

Mon, 15 Dec 2025 03:00:00 +0000

Microsoft Dataverse with Python and Zato Services

2025-12-15, by Dariusz Suchojad

Overview

Microsoft Dataverse is a cloud-based data storage and management platform, often used with PowerApps and Dynamics 365.

Integrating Dataverse with Python via Zato enables automation, API orchestration, and seamless CRUD (Create, Read, Update, Delete) operations on any Dataverse object.

Below, you'll find practical code examples for working with Dataverse from Python, including detailed comments and explanations. The focus is on the "accounts" entity, but the same approach applies to any object in Dataverse.

Connecting to Dataverse and retrieving accounts

The main service class configures the Dataverse client and retrieves all accounts. Both the handle and get_accounts methods are shown together for clarity.

# -*- coding: utf-8 -*-

# Zato
from zato.common.typing_ import any_
from zato.server.service import DataverseClient, Service

class MyService(Service):

    def handle(self):

        # Set up Dataverse credentials - in a real service,
        # this would go to your configuration file.

        tenant_id = '221de69a-602d-4a0b-a0a4-1ff2a3943e9f'
        client_id = '17aaa657-557c-4b18-95c3-71d742fbc6a3'
        client_secret = 'MjsrO1zc0.WEV5unJCS5vLa1'
        org_url = 'https://org123456.api.crm4.dynamics.com'

        # Build the Dataverse client using the credentials
        client = DataverseClient(
            tenant_id=tenant_id,
            client_id=client_id,
            client_secret=client_secret,
            org_url=org_url
        )

        # Retrieve all accounts using a helper method
        accounts = self.get_accounts(client)

        # Process the accounts as needed (custom logic goes here)
        pass

    def get_accounts(self, client:'DataverseClient') -> 'any_':

        # Specify the API path for the accounts entity
        path = 'accounts'

        # Call the Dataverse API to retrieve all accounts
        response = client.get(path)

        # Log the response for debugging/auditing

        self.logger.info(f'Dataverse response (get accounts): {response}')

        # Return the API response to the caller
        return response

{'@odata.context': 'https://org1234567.crm4.dynamics.com/api/data/v9.0/$metadata#accounts',
'value': [{'@odata.etag': 'W/"11122233"', 'territorycode': 1,
'accountid': 'd92e6f18-36fb-4fa8-b7c2-ecc7cc28f50c', 'name': 'Zato Test Account 1',
'_owninguser_value': 'ea4dd84c-dee6-405d-b638-c37b57f00938'}]}

Let's check more examples - you'll note they all follow the same pattern as the first one.

Retrieving an Account by ID

def get_account_by_id(self, client:'DataverseClient', account_id:'str') -> 'any_':

    # Construct the API path using the account's GUID
    path = f'accounts({account_id})'

    # Call the Dataverse API to fetch the account
    response = client.get(path)

    # Log the response for traceability
    self.logger.info(f'Dataverse response (get account by ID): {response}')

    # Return the fetched account
    return response

Retrieving an account by name

def get_account_by_name(self, client:'DataverseClient', account_name:'str') -> 'any_':

    # Construct the API path with a filter for the account name
    path = f"accounts?$filter=name eq '{account_name}'"

    # Call the Dataverse API with the filter
    response = client.get(path)

    # Log the response for auditing
    self.logger.info(f'Dataverse response (get account by name): {response}')

    # Return the filtered account(s)
    return response

Creating a new account

def create_account(self, client:'DataverseClient') -> 'any_':

    # Specify the API path for account creation
    path = 'accounts'

    # Prepare the data for the new account
    account_data = {
        'name': 'New Test Account',
        'telephone1': '+1-555-123-4567',
        'emailaddress1': 'hello@example.com',
        'address1_city': 'Prague',
        'address1_country': 'Czech Republic',
    }

    # Call the Dataverse API to create the account
    response = client.post(path, account_data)

    # Log the response for traceability
    self.logger.info(f'Dataverse response (create account): {response}')

    # Return the API response
    return response

Updating an existing account

def update_account(self, client:'DataverseClient', account_id:'str') -> 'any_':

    # Prepare the data to update
    update_data = {
        'name': 'Updated Account Name',
        'telephone1': '+1-555-987-6543',
        'emailaddress1': 'hello2@example.com',
    }

    # Call the Dataverse API to update the account by ID
    response = client.patch(f'accounts({account_id})', update_data)

    # Log the response for auditing
    self.logger.info(f'Dataverse response (update account): {response}')

    # Return the updated account response
    return response

Deleting an Account

def delete_account(self, client:'DataverseClient', account_id:'str') -> 'any_':

    # Call the Dataverse API to delete the account
    response = client.delete(f'accounts({account_id})')

    # Log the response for traceability
    self.logger.info(f'Dataverse response (delete account): {response}')

    # Return the API response
    return response

API path vs. PowerApps UI table names

A detail to note when working with Dataverse APIs is that the names you see in the PowerApps or Dynamics UI are not always the same as the paths expected by the API. For example:

In PowerApps, you may see a table called Account.
In the API, you must use the path accounts (lowercase, plural) when making requests.

This pattern applies to all Dataverse objects: always check the API documentation or inspect the metadata to determine the correct entity path.

Working with other Dataverse objects

While the examples above focus on the "accounts" entity, the same approach applies to any object in Dataverse: contacts, leads, opportunities, custom tables, and more. Simply adjust the API path and payload as needed.

Full CRUD Support

With Zato and Python, you get full CRUD (Create, Read, Update, Delete) capability for any Dataverse entity. The methods shown above can be adapted for any object, allowing you to automate, integrate, and orchestrate data flows across your organization.

Summary

This article has shown how to connect to Microsoft Dataverse from Python using Zato, perform CRUD operations, and understand the mapping between UI and API paths. These techniques enable robust integration and automation scenarios with any Dataverse data.

More resources

➤ Microsoft 365 APIs and Python Tutorial
➤ Python API integration tutorials
➤ What is an integration platform?
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?
➤ Open-source iPaaS in Python

Automate Microsoft 365 like a pro: skip the OAuth headaches

Mon, 08 Dec 2025 03:00:00 +0000

Automate Microsoft 365 like a pro: skip the OAuth headaches

2025-12-08, by Dariusz Suchojad

Are you tired of fighting with Microsoft 365 APIs and complex authentication flows? This practical tutorial cuts through the complexity, showing you how to build production-ready Python automations for Outlook, Calendar, and SharePoint in just 60 minutes.

No OAuth2 struggles, no Graph API complexity - just working code you can deploy today.

What you'll learn in the tutorial:

Connect to Microsoft 365 APIs with minimal Python code
Build real email, calendar, and SharePoint automations
Schedule background tasks without programming
Access Entra ID (Azure AD) user data effortlessly
Implement enterprise-grade patterns that solve actual business problems

More resources

Network packet broker automation in Python

Mon, 01 Dec 2025 03:00:00 +0000

Network packet broker automation in Python

2025-12-01, by Dariusz Suchojad

Packet brokers are crucial for network engineers, providing a clear, detailed view of network traffic, aiding in efficient issue identification and resolution.

But what is a network packet broker (NBP) really? Why are they needed? And how to automate one in Python?

➤ Read this article about network packet brokers and their automation in Python to find out more.

More resources

➤ Click here to read more about using Python and Zato in telecommunications
➤ Python API integration tutorials
➤ What is an integration platform?
➤ Open-source iPaaS in Python

Automation in Python

Mon, 24 Nov 2025 03:00:00 +0000

Automation in Python

2025-11-24, by Dariusz Suchojad

How to automate systems in Python and how the Zato Python integration platform differs from a network automation tool, how to start using it, along with a couple of examples of integrations with Office 365 and Jira, is what the latest article is about.

➤ Read it here: Systems Automation in Python.

More resources

Enterprise gRPC Services with Python and Zato

Tue, 22 Jul 2025 03:00:00 +0000

Enterprise gRPC Services with Python and Zato

2025-07-22, by Dariusz Suchojad

Overview

gRPC is a high-performance framework for building distributed systems and microservices. It leverages Protocol Buffers for serialization and HTTP/2 for transport, making it ideal for production environments requiring low-latency communication.

This Python integration with Zato demonstrates how to build scalable gRPC services with strongly-typed schemas. The approach enables efficient API orchestration and reliable data exchange between services using type-safe contracts.

Below you'll find a Protocol Buffer schema definition and an enterprise-ready API integration service showing end-to-end implementation.

Protocol buffer schema definition

First, define your gRPC service contract using Protocol Buffers. This example shows a mandate registration service - a financial services process where a customer authorizes a merchant or service provider to automatically collect recurring payments from their bank account.

syntax = "proto3";

package debit_mandate;

service MandateRegistrationService {
    rpc RegisterMandate (MandateRegistrationRequest) returns (MandateActionResponse) {}
}

enum MandateCollectionFrequencyType {
    MANDATE_COLLECTION_FREQUENCY_TYPE_UNSPECIFIED = 0;
    MANDATE_COLLECTION_FREQUENCY_TYPE_DAILY = 1;
    MANDATE_COLLECTION_FREQUENCY_TYPE_WEEKLY = 2;
    MANDATE_COLLECTION_FREQUENCY_TYPE_MONTHLY = 3;
}

enum CurrencyCode {
    CURRENCY_CODE_UNSPECIFIED = 0;
    CURRENCY_CODE_EUR = 1;
}

message MoneyObject {
    CurrencyCode currency_code = 1;
    bool positive_value = 4;
    uint32 units = 2;
    uint32 nanos = 3;
}

message MandateRegistrationRequest {
    string debtor_name = 1;
    MandateCollectionFrequencyType collection_frequency = 2;
    MoneyObject instalment_amount = 3;
    string debtor_identity_number = 4;
    string bank_account_number = 5;
}

message MandateActionResponse {
    uint32 response_code = 1;
    string response_description = 2;
}

Enabling gRPC in Zato Projects

To use gRPC in your integration services, add the required dependencies to your project's requirements file:

# config/python-reqs/requirements.txt
grpcio>=1.50.0
grpcio-tools>=1.50.0
protobuf>=4.21.0

When your container starts, these dependencies will be automatically installed and available to your service.

Zato service integration

Create a Zato service that uses the gRPC client with proper input handling:

# -*- coding: utf-8 -*-

# stdlib
from dataclasses import dataclass

# Zato
from zato.server.service import Model, Service

# gRPC Code
from debit_mandate_pb2 import MANDATE_COLLECTION_FREQUENCY_TYPE_MONTHLY
from debit_mandate_client import MandateClient

# ###########################################################################

@dataclass(init=False)
class MandateRegistrationRequest(Model):
    debtor_name: str
    debtor_identity_number: str
    bank_account_number: str
    collection_frequency: int
    amount_units: int

@dataclass(init=False)
class MandateRegistrationResponse(Model):
    status: int
    details: str

# ###########################################################################

class MandateRegistrationService(Service):

    # Use API models
    input = MandateRegistrationRequest
    output = MandateRegistrationResponse

    def handle(self):

        # Local variables
        input = self.request.input

        # Get a client for the gRPC endpoint ..
        client = MandateClient()

        try:
            # .. invoke the endpoint ..
            grpc_response = client.register_mandate(
                debtor_name=input.debtor_name,
                debtor_identity_number=input.debtor_identity_number,
                bank_account_number=input.bank_account_number,
                collection_frequency=input.collection_frequency,
                amount_units=input.amount_units,
            )

            # .. map its response to ours ..
            response = MandateRegistrationResponse()
            response.status = grpc_response.response_code
            response.details = grpc_response.response_description

            # .. and return it to the caller.
            self.response.payload = response

        finally:
            client.close()

Testing the Integration

Once deployed, you can invoke your Zato service through a REST endpoint:

curl -X POST http://localhost:17010/api/mandate/register \
  -H "Content-Type: application/json" \
  -d '{
    "debtor_name": "John Doe",
    "debtor_identity_number": "123456789",
    "bank_account_number": "GB29NWBK60161331926819",
    "collection_frequency": 3,
    "amount_units": 100
  }'

The service returns a structured JSON response:

{"status": 200, "details": "Mandate registered successfully"}

More resources

Enterprise Python: integrating with Salesforce

Mon, 02 Jun 2025 03:00:00 +0000

Enterprise Python: integrating with Salesforce

2025-06-02, by Dariusz Suchojad

Overview

Salesforce connections are one of the newest additions to Zato 3.2, allowing you to look up and manage Salesforce records and other business data. To showcase it, the article will create a sample Salesforce marketing campaign in a way that does not require the usage of anything else except for basic REST APIs combined with plain Python objects, such as dicts.

If you have not done it already, you can download Zato here.

Basic workflow

The scope of our works will be:

Creating Salesforce credentials for our integration project
Defining a Salesforce connection in Zato
Authoring a service that will map input data to the format that Salesforce expects
Creating a Zato REST channel that will be invoked through curl during tests
Testing the integration

Creating Salesforce credentials

To be able to create as connection to Salesforce in the next step, we need a few credentials. There is a full article about how to prepare them and this section is the gist of it.

In runtime, based on this information, Zato will obtain the necessary authentication and authorization tokens itself, which means that you will only focus on the business side of the integrations, not on the low-level aspects of it.

The process of obtaining the credentials needs to be coordinated with an administrator of your organization. To assist in that, the screenshots below explain where to find them.

The credentials are:

Username and password
Consumer key
Consumer secret

The username and password are simply the same credentials that can be used to log in to Salesforce:

Consumer key and secret are properties of a connected app - this is a term that Salesforce uses for API clients that invoke its services. If you are already an experienced Salesforce REST API user, you may know the key and secret under their aliases of "client_id" and "client_secret" - these are the same objects.

Note that when a connected app already exists and you would like to retrieve the key and secret, they will be available under the "View" menu option for the app, not under "Edit" or "Manage".

Defining a Salesforce connection in Zato

With all the credentials in place, we can create a new Salesforce connection in Zato Dashboard, as below.

Authoring an integration service in Python

Above, we created a connection definition that lets Zato obtain session tokens and establish connections to Salesforce. Now, we can create an API service that will make use of such connections.

In the example below, we are using the POST REST method to invoke an endpoint that creates new Salesforce campaigns. In your own integrations, you can invoke any other Salesforce endpoint, using any REST method as needed, by following the same pattern, which is, create a model with input fields, build a Python dict for the request to Salesforce, invoke it and map all the required from the response from Salesforce to that which your own service returns to its own callers.

Note that we use a datamodel-based SimpleIO definition for the service. Among other things, although we are not going to do it here, this would let us offer definitions for this and other services.

# -*- coding: utf-8 -*-

# stdlib
from dataclasses import dataclass

# Zato
from zato.server.service import Model, Service

# ###########################################################################

if 0:
    from zato.server.connection.salesforce import SalesforceClient

# ###########################################################################

@dataclass(init=False)
class CreateCampaignRequest(Model):
    name:    str
    segment: str

# ###########################################################################

@dataclass(init=False)
class CreateCampaignResponse(Model):
    campaign_id: str

# ###########################################################################

class CreateCampaign(Service):

    class SimpleIO:
        input  = CreateCampaignRequest
        output = CreateCampaignResponse

    def handle(self):

        # This is our input data
        input = self.request.input # type: CreateCampaignRequest

        # Salesforce REST API endpoint to invoke - note that Zato
        # will add a prefix to it containing the API version.
        path = '/sobjects/Campaign/'

        # Build the request to Salesforce based on what we received
        request = {
          'Name': input.name,
          'Segment__c': input.segment,
        }

        # .. create a reference to our connection definition ..
        salesforce = self.cloud.salesforce['My Salesforce Connection']

        # .. obtain a client to Salesforce ..
        with salesforce.conn.client() as client: # type: SalesforceClient

            # .. create the campaign now ..
            sf_response = client.post(path, request)

        # .. build our response object ..
        response = CreateCampaignResponse()
        response.campaign_id = sf_response['id']

        # .. and return its ID to our caller.
        self.response.payload = response

# ###########################################################################

Creating a REST channel

Note that we assign HTTP Basic Auth credentials to the channel. In this manner, it is possible for clients of this REST channel to authenticate using a method that they are already familiar which simplifies everyone's work - it is Zato that deals with how to authenticate against Salesforce whereas your API clients use the ubiquitous HTTP Basic Auth method.

Testing

The last step is to invoke the newly created channel:

$ curl http://api:password@localhost:17010/api/campaign/create -d '{"name":"Hello", "segment":"123"}'
{"campaign_id":"8901Z3VHXDTebEJWs"}
$

That is everything - you have just integrated with Salesforce and exposed a REST channel for external applications to integrate with!

More resources

Sustainable water management with IoT, Open Source and Python

Tue, 29 Apr 2025 03:00:00 +0000

Sustainable water management with IoT, Open Source and Python

2025-04-29, by Dariusz Suchojad

"Towards a cyber-physical system for sustainable and smart building: a use case for optimizing water consumption on a SmartCampus" is an interesting, Open Access paper via Springer Publishing authored by Sergio Barroso Ramírez, Pablo Bustos and Pedro Núñez Trujillo of Universidad de Extremadura uses Python, Zato and other open source technologies for IoT integrations.

More resources

Integrating with Jira APIs

Thu, 24 Apr 2025 03:00:00 +0000

Integrating with Jira APIs

2025-04-24, by Dariusz Suchojad

Overview

Continuing in the series of articles about newest cloud connections in Zato 4.1, this episode covers Atlassian Jira from the perspective of invoking its APIs to build integrations between Jira and other systems.

There are essentially two use modes of integrations with Jira:

Jira reacts to events taking place in your projects and invokes your endpoints accordingly via WebHooks. In this case, it is Jira that explicitly establishes connections with and sends requests to your APIs.
Jira projects are queried periodically or as a consequence of events triggered by Jira using means other than WebHooks.

The first case is usually more straightforward to conceptualize - you create a WebHook in Jira, point it to your endpoint and Jira invokes it when a situation of interest arises, e.g. a new ticket is opened or updated. I will talk about this variant of integrations with Jira in a future instalment as the current one is about the other situation, when it is your systems that establish connections with Jira.

The reason why it is more practical to first speak about the second form is that, even if WebHooks are somewhat easier to reason about, they do come with their own ramifications.

To start off, assuming that you use the cloud-based version of Jira (e.g. https://example.atlassian.net), you need to have a publicly available endpoint for Jira to invoke through WebHooks. Very often, this is undesirable because the systems that you need to integrate with may be internal ones, never meant to be exposed to public networks.

Secondly, your endpoints need to have a TLS certificate signed by a public Certificate Authority and they need to be accessible on port 443. Again, both of these are something that most enterprise systems will not allow at all or it may take months or years to process such a change internally across the various corporate departments involved.

Lastly, even if a WebHook can be used, it is not always a given that the initial information that you receive in the request from a WebHook will already contain everything that you need in your particular integration service. Thus, you will still need a way to issue requests to Jira to look up details of a particular object, such as tickets, in this way reducing WebHooks to the role of initial triggers of an interaction with Jira, e.g. a WebHook invokes your endpoint, you have a ticket ID on input and then you invoke Jira back anyway to obtain all the details that you actually need in your business integration.

The end situation is that, although WebHooks are a useful concept that I will write about in a future article, they may very well not be sufficient for many integration use cases. That is why I start with integration methods that are alternative to WebHooks.

Alternatives to WebHooks

If, in our case, we cannot use WebHooks then what next? Two good approaches are:

Scheduled jobs
Reacting to emails (via IMAP)

Scheduled jobs will let you periodically inquire with Jira about the changes that you have not processed yet. For instance, with a job definition as below:

Now, the service configured for this job will be invoked once per minute to carry out any integration works required. For instance, it can get a list of tickets since the last time it ran, process each of them as required in your business context and update a database with information about what has been just done - the database can be based on Redis, MongoDB, SQL or anything else.

Integrations built around scheduled jobs make most sense when you need to make periodic sweeps across a large swaths of business data, these are the "Give me everything that changed in the last period" kind of interactions when you do not know precisely how much data you are going to receive.

In the specific case of Jira tickets, though, an interesting alternative may be to combine scheduled jobs with IMAP connections:

The idea here is that when new tickets are opened, or when updates are made to existing ones, Jira will send out notifications to specific email addresses and we can take advantage of it.

For instance, you can tell Jira to CC or BCC an address such as zato@example.com. Now, Zato will still run a scheduled job but instead of connecting with Jira directly, that job will look up unread emails for it inbox ("UNSEEN" per the relevant RFC).

Anything that is unread must be new since the last iteration which means that we can process each such email from the inbox, in this way guaranteeing that we process only the latest updates, dispensing with the need for our own database of tickets already processed. We can extract the ticket ID or other details from the email, look up its details in Jira and the continue as needed.

All the details of how to work with IMAP emails are provided in the documentation but it would boil down to this:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class MyService(Service):

    def handle(self):
        conn = self.email.imap.get('My Jira Inbox').conn

        for msg_id, msg in conn.get():

            # Process the message here ..
            process_message(msg.data)

            # .. and mark it as seen in IMAP.
            msg.mark_seen()

The natural question is - how would the "process_message" function extract details of a ticket from an email?

There are several ways:

Each email has a subject of a fixed form - "[JIRA] (ABC-123) Here goes description". In this case, ABC-123 is the ticket ID.
Each email will contain a summary, such as the one below, which can also be parsed:

         Summary: Here goes description
             Key: ABC-123
             URL: https://example.atlassian.net/browse/ABC-123
         Project: My Project
      Issue Type: Improvement
Affects Versions: 1.3.17
     Environment: Production
        Reporter: Reporter Name
        Assignee: Assignee Name

Finally, each email will have an "X-Atl-Mail-Meta" header with interesting metadata that can also be parsed and extracted:

X-Atl-Mail-Meta: user_id="123456:12d80508-dcd0-42a2-a2cd-c07f230030e5",
                 event_type="Issue Created",
                 tenant="https://example.atlassian.net"

The first option is the most straightforward and likely the most convenient one - simply parse out the ticket ID and call Jira with that ID on input for all the other information about the ticket. How to do it exactly is presented in the next chapter.

Regardless of how we parse the emails, the important part is that we know that we invoke Jira only when there are new or updated tickets - otherwise there would not have been any new emails to process. Moreover, because it is our side that invokes Jira, we do not expose our internal system to the public network directly.

However, from the perspective of the overall security architecture, email is still part of the attack surface so we need to make sure that we read and parse emails with that in view. In other words, regardless of whether it is Jira invoking us or our reading emails from Jira, all the usual security precautions regarding API integrations and accepting input from external resources, all that still holds and needs to be part of the design of the integration workflow.

Creating Jira connections

The above presented the ways in which we can arrive at the step of when we invoke Jira and now we are ready to actually do it.

As with other types of connections, Jira connections are created in Zato Dashboard, as below. Note that you use the email address of a user on whose behalf you connect to Jira but the only other credential is that user's API token previously generated in Jira, not the user's password.

Invoking Jira

With a Jira connection in place, we can now create a Python API service. In this case, we accept a ticket ID on input (called "a key" in Jira) and we return a few details about the ticket to our caller.

This is the kind of a service that could be invoked from a service that is triggered by a scheduled job. That is, we would separate the tasks, one service would be responsible for opening IMAP inboxes and parsing emails and the one below would be responsible for communication with Jira.

Thanks to this loose coupling, we make everything much more reusable - that the services can be changed independently is but one part and the more important side is that, with such separation, both of them can be reused by future services as well, without tying them rigidly to this one integration alone.

# -*- coding: utf-8 -*-

# stdlib
from dataclasses import dataclass

# Zato
from zato.common.typing_ import cast_, dictnone
from zato.server.service import Model, Service

# ###########################################################################

if 0:
    from zato.server.connection.jira_ import JiraClient

# ###########################################################################

@dataclass(init=False)
class GetTicketDetailsRequest(Model):
    key: str

@dataclass(init=False)
class GetTicketDetailsResponse(Model):
    assigned_to: str = ''
    progress_info: dictnone = None

# ###########################################################################

class GetTicketDetails(Service):

    class SimpleIO:
        input  = GetTicketDetailsRequest
        output = GetTicketDetailsResponse

    def handle(self):

        # This is our input data
        input = self.request.input # type: GetTicketDetailsRequest

        # .. create a reference to our connection definition ..
        jira = self.cloud.jira['My Jira Connection']

        # .. obtain a client to Jira ..
        with jira.conn.client() as client:

            # Cast to enable code completion
            client = cast_('JiraClient', client)

            # Get details of a ticket (issue) from Jira
            ticket = client.get_issue(input.key)

        # Observe that ticket may be None (e.g. invalid key), hence this 'if' guard ..
        if ticket:

            # .. build a shortcut reference to all the fields in the ticket ..
            fields = ticket['fields']

            # .. build our response object ..
            response = GetTicketDetailsResponse()
            response.assigned_to = fields['assignee']['emailAddress']
            response.progress_info = fields['progress']

            # .. and return the response to our caller.
            self.response.payload = response

# ###########################################################################

Creating a REST channel and testing it

The last remaining part is a REST channel to invoke our service through. We will provide the ticket ID (key) on input and the service will reply with what was found in Jira for that ticket.

We are now ready for the final step - we invoke the channel, which invokes the service which communicates with Jira, transforming the response from Jira to the output that we need:

$ curl localhost:17010/jira1 -d '{"key":"ABC-123"}'
{
    "assigned_to":"zato@example.com",
    "progress_info": {
        "progress": 10,
        "total": 30
    }
}
$

And this is everything for today - just remember that this is just one way of integrating with Jira. The other one, using WebHooks, is something that I will go into in one of the future articles.

More resources

Airport Integrations in Python

Thu, 10 Apr 2025 03:00:00 +0000

Airport Integrations in Python

2025-04-10, by Dariusz Suchojad

Did you know you can use Python as an integration platform for your airport systems? It's Open Source too.

From AODB, transportation, business operations and partner networks, to IoT, cloud and hybrid deployments, you can now use Python to build flexible, scalable and future-proof architectures that integrate your airport systems and support your master plan.

➤ Read here about what is possible and learn more why Python and Open Source are the right choice.

More resources

➤ Python API integration tutorials
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?
➤ Open-source iPaaS in Python
➤ What is a Network Packet Broker? How to automate networks in Python?

How to correctly integrate APIs in Python

Mon, 17 Mar 2025 03:00:00 +0000

How to correctly integrate APIs in Python

2025-03-17, by Dariusz Suchojad

Understanding how to effectively integrate various systems and APIs is crucial. Yet, without a dedicated integration platform, the result will be brittle point-to-point, spaghetti integrations, that never lead to good outcomes.

➤ Read this article about Zato, an open-source integration platform in Python, for an overview of what to avoid and how to do it correctly instead.

More resources

LDAP and Active Directory as Python API Services

Sun, 19 Jan 2025 03:00:00 +0000

LDAP and Active Directory as Python API Services

2025-01-19, by Dariusz Suchojad

LDAP and Active Directory often play key a role in the management of a company's network resources yet it is not always very convenient to query a directory directly using the LDAP syntax and protocol that few people truly specialize in. This is why in this article we are using Zato to offer a REST API on top of directory services so that API clients can use REST and JSON instead.

Installing Zato

Start off by installing Zato - if you are not sure what to choose, pick the Docker Quickstart option and this will set up a working environment in a few minutes.

Creating connections

Once Zato is running, connections can be easily created in its Dashboard (by default, http://127.0.0.1:8183). Navigate to Connections -> Outgoing -> LDAP ..

.. and then click Create a new connection which will open a form as below:

The same form works for both regular LDAP and Active Directory - in the latter case, make sure that Auth type is set to NTLM.

The most important information is:

User credentials
Authentication type
Server or servers to connect to

Note that if authentication type is not NTLM, user credentials can be provided using the LDAP syntax, e.g. uid=MyUser,ou=users,o=MyOrganization,dc=example,dc=com.

Right after creating a connection be sure to set its password too - the password asigned by default is a randomly generated one.

Pinging

It is always prudent to ping a newly created connection to ensure that all the information entered was correct.

Note that if you have more than one server in a pool then the first available one of them will be pinged - it is the whole pool that is pinged, not a particular part of it.

Active Directory as a REST service

As the first usage example, let's create a service that will translate JSON queries into LDAP lookups - given username or email the service will basic information about the person's account, such as first and last name.

Note that the conn object returned by client.get() below is capable of running any commands that its underlying Python library offers - in this case we are only using searches but any other operation can also be used, e.g. add or modify as well.

# -*- coding: utf-8 -*-

# stdlib
from json import loads

# Bunch
from bunch import bunchify

# Zato
from zato.server.service import Service

# Where in the directory we expect to find the user
search_base = 'cn=users, dc=example, dc=com'

# On input, we are looking users up by either username or email
search_filter = '(&(|(uid={user_info})(mail={user_info})))'

# On output, we are interested in username, first name, last name and the person's email
query_attributes = ['uid', 'givenName', 'sn', 'mail']

class ADService(Service):
    """ Looks up users in AD by their username or email.
    """
    class SimpleIO:
        input_required = 'user_info'
        output_optional = 'message', 'username', 'first_name', 'last_name', 'email'
        response_elem = None
        skip_empty_keys = True

    def handle(self):

        # Connection name to use
        conn_name = 'My AD Connection'

        # Get a handle to the connection pool
        with self.out.ldap[conn_name].conn.client() as client:

            # Get a handle to a particular connection
            with client.get() as conn:

                # Build a filter to find a user by
                user_info = self.request.input['user_info']
                user_filter = search_filter.format(user_info=user_info)

                # Returns True if query succeeds and has any information on output
                if conn.search(search_base, user_filter, attributes=query_attributes):

                    # This is where the actual response can be found
                    response = conn.entries

                    # In this case, we expect at most one user matching input criteria
                    entry = response[0]

                    # Convert it to JSON for easier handling ..
                    entry = entry.entry_to_json()

                    # .. and load it from JSON to a Python dict
                    entry = loads(entry)

                    # Convert to a Bunch instance to get dot access to dictionary keys
                    entry = bunchify(entry['attributes'])

                    # Now, actually produce a JSON response. For simplicity's sake,
                    # assume that users have only one of email or other attributes.
                    self.response.payload.message = 'User found'
                    self.response.payload.username = entry.uid[0]
                    self.response.payload.first_name = entry.givenName[0]
                    self.response.payload.last_name = entry.sn[0]
                    self.response.payload.email = entry.mail[0]

                else:
                    # No business response = no such user found
                    self.response.payload.message = 'No such user'

After creating a REST channel, we can invoke the service from command line, thus confirming that we can offer the directory as a REST service:

$ curl "localhost:11223/api/get-user?user_info=MyOrganization\\MyUser" ; echo
{
    "message": "User found",
    "username": "MyOrganization\\MyUser",
    "first_name": "First",
    "last_name": "Last",
    "email": "address@example.com"
}
$

More resources

Understanding API rate-limiting techniques

Mon, 13 Jan 2025 03:00:00 +0000

Understanding API rate-limiting techniques

2025-01-13, by Dariusz Suchojad

Enabling rate-limiting in Zato means that access to Zato APIs can be throttled per endpoint, user or service - including options to make limits apply to specific IP addresses only - and if limits are exceeded within a selected period of time, the invocation will fail. Let's check how to use it all.

API rate limiting works on several levels and the configuration is always checked in the order below, which follows from the narrowest, most specific parts of the system (endpoints), through users which may apply to multiple endpoints, up to services which in turn may be used by both multiple endpoints and users.

First, per-endpoint limits
Then, per-user limits
Finally, per-service limits

When a request arrives through an endpoint, that endpoint's rate limiting configuration is checked. If the limit is already reached for the IP address or network of the calling application, the request is rejected.

Next, if there is any user associated with the endpoint, that account's rate limits are checked in the same manner and, similarly, if they are reached, the request is rejected.

Finally, if the endpoint's underlying service is configured to do so, it also checks if its invocation limits are not exceeded, rejecting the message accordingly if they are.

Note that the three levels are distinct yet they overlap in what they allow one to achieve.

For instance, it is possible to have the same user credentials be used in multiple endpoints and express ideas such as "Allow this and that user to invoke my APIs 1,000 requests/day but limit each endpoint to at most 5 requests/minute no matter which user".

Moreover, because limits can be set on services, it is possible to make it even more flexible, e.g. "Let this service be invoked at most 10,000 requests/hour, no matter which user it is, with particular users being able to invoke at most 500 requests/minute, no matter which service, topping it off with per separate limits for REST vs. SOAP vs. JSON-RPC endpoint, depending on what application is invoke the endpoints". That lets one conveniently express advanced scenarios that often occur in practical situations.

Also, observe that API rate limiting applies to REST, SOAP and JSON-RPC endpoints only, it is not used with other API endpoints, such as AMQP, IBM MQ, SAP, task scheduler or any other technologies. However, per-service limits work no matter which endpoint the service is invoked with and they will work with endpoints such as WebSockets, ZeroMQ or any other.

Lastly, limits pertain to with incoming requests only - any outgoing ones, from Zato to external resources - are not covered by it.

Per-IP restrictions

The architecture is made even more versatile thanks to the fact that for each object - endpoint, user or service - different limits can be configured depending on the caller's IP address.

This adds yet another dimension and allows to express ideas commonly witnessed in API-based projects, such as:

External applications, depending on their IP addresses, can have their own limits
Internal users, e.g. employees of the company using VPN, may have hire limits if their addresses are in the 172.x.x.x range
For performance testing purposes, access to Zato from a few selected hosts may have no limits at all

IP-based limits work hand in hand are an integral part of the mechanism - they do not rule out per-endpoit, user or service limits. In fact, for each such object, multiple IP-using limits can be set independently, thus allowing for highest degree of flexibility.

Exact or approximate

Rate limits come in two types:

Exact
Approximate

Exact rate limits are just that, exact - they en that a limit is not exceeded at all, not even by a single request.

Approximate limits may let a very small number of requests to exceed the limit with the benefit being that approximate limits are faster to check than exact ones.

When to use which type depends on a particular project:

In some projects, it does not really matter if callers have a limit of 1,000 requests/minute or 1,005 requests/minute because the difference is too tiny to make a business impact. Approximate limits work best in this case.
In other projects, there may be requirements that the limit never be exceeded no matter the circumstances. Use exact limits here.

Python code and web-admin

Alright, let's check how to define the limits in the Zato Dashboard. We will use the sample service below:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class Sample(Service):
    name = 'api.sample'

    def handle(self):

        # Return a simple string on response
        self.response.payload = 'Hello there!\n'

Now, in web-admin, we will configure limits - separately for the service, a new and a new REST API channel (endpoint).

Points of interest:

Configuration for each type of object is independent - within the same invocation some limits may be exact, some may be approximate
There can be multiple configuration entries for each object
A unit of time is "m", "h" or "d", depending on whether the limit is per minute, hour or day, respectively
All limits within the same configuration are checked in the order of their definition which is why the most generic ones should be listed first

Testing it out

Now, all is left is to invoke the service from curl.

As long as limits are not reached, a business response is returned:

$ curl http://my.user:password@localhost:11223/api/sample
Hello there!
$

But if a limit is reached, the caller receives an error message with the 429 HTTP status.

$ curl -v http://my.user:password@localhost:11223/api/sample
*   Trying 127.0.0.1...

...

< HTTP/1.1 429 Too Many Requests
< Server: Zato
< X-Zato-CID: b8053d68612d626d338b02

...

{"zato_env":{"result":"ZATO_ERROR","cid":"b8053d68612d626d338b02eb",
 "details":"Error 429 Too Many Requests"}}
$

Note that the caller never knows what the limit was - that information is saved in Zato server logs along with other details so that API authors can correlate what callers get with the very rate limiting definition that prevented them from accessing the service.

zato.common.rate_limiting.common.RateLimitReached: Max. rate limit of 100/m reached;
from:`10.74.199.53`, network:`*`; last_from:`127.0.0.1;
last_request_time_utc:`2025-01-12T15:30:41.943794;
last_cid:`5f4f1ef65490a23e5c37eda1`; (cid:b8053d68612d626d338b02)

And this is it - we have created a new API rate limiting definition in Zato and tested it out successfully!

More resources

Network automation getting started

Tue, 07 Jan 2025 03:00:00 +0000

Network automation getting started

2025-01-07, by Dariusz Suchojad

All network engineers and architects understand that automation is essential for efficiency and reliability, especially with the increasing shift to cloud-based environments, where it's no longer sufficient to merely connect network elements alone, without taking the bigger IT picture into account.

The latest article about network automation will introduce you to a Python-based integration platform designed to automate network tasks, connect diverse systems, and manage complex workflows.

You'll see not only how to automate a network device, but also how to connect cloud applications outside of what a typical automation tool would do, including Jira for tickets and Microsoft 365 to manage email, all using basic Python code.

Read the article here: Network Automation in Python.

More resources

Using OAuth in API Integrations

Mon, 23 Dec 2024 03:00:00 +0000

Using OAuth in API Integrations

2024-12-23, by Dariusz Suchojad

OAuth is often employed in processes requiring permissions to be granted to frontend applications and end users. Yet, what we typically need in API systems integrations is a way to secure connections between the integration middleware and backend systems without a need for any ongoing human interactions.

OAuth can be a good choice for that scenario and this article shows how it can be achieved in Python, with backend systems using REST and HL7 FHIR.

What we would like to have

Let's say we have a typical integration scenario as in the diagram below:

External systems and applications invoke the interoperability layer (Zato) which is expected to further invoke a few backend systems, e.g. a REST and HL7 FHIR one so as to return a combined result of backend API invocations. It does not matter what technology the client systems use, i.e. whether they are REST ones or not.
The interoperability layer needs to identify itself with the backend systems before it is allowed to invoke them - they need to make sure that it really is Zato and that it accesses only the resources allowed.
An OAuth server issues time-based access tokens, which are simple strings, like web browser session cookies, confirming that such and such bearer of the said token is allowed to make such and such requests. Note that the tokens have an explicit expiration time, e.g. they will become invalid after one hour. Also observe that Zato stores the tokens as-is, they are genuinely opaque strings.
If a client system invokes the interoperability layer, the layer will obtain a token from the OAuth server and keep it in an internal cache. Next, Zato will invoke the backend systems, bearing the token among other HTTP headers. Each invoked backend system will extract the token from the incoming request and validate it.

How the validation looks like in practices is something that Zato will not be aware of because it treats the token as an opaque string but, in practice, if the token is self-contained (e.g. JWT data) the system may validate it on its own, and if it is not self-contained, the system may invoke an introspection endpoint on the OAuth server to validate the access token from Zato.

Once the validation succeeds, the backend system will reply with the business data and the interoperability layer will combine the results for the calling application's benefit.

In subsequent requests, the same access token will be reused by Zato with the same flow of messages as previously. However, if the cached token expires, Zato will request a new one from the OAuth server - this will be transparent to the calling application - and the flow will resume.

In OAuth terminology, what is described above has specific names, the overall flow of messages between Zato and the OAuth server is called a "Client Credential Flow" and Zato is then considered a "client" from the OAuth server's perspective.

Configuring OAuth

First, we need to create an OAuth security definition that contains the OAuth server's connection details. In this case, the server is Okta. Note the scopes field - it is a list of permissions ("scopes") that Zato will be able to make use of.

What exactly the list of scopes should look like is something to be coordinated with the people who are responsible for the configuration of the OAuth server. If it is you personally, simply ensure that what is in the the OAuth server and in Zato is in sync.

Calling REST

To invoke REST services, fill out a form as below, pointing the "Security" field to the newly created OAuth definition. This suffices for Zato to understand when and how to obtain new tokens from the underlying OAuth server.

Here is sample code to invoke a backend REST system - note that we merely refer to a connection by its name, without having to think about security at all. It is Zato that knows how to get and use OAuth tokens as required.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class GetClientBillingPlan(Service):
    """ Returns a billing plan for the input client.
    """
    def handle(self):

        # In a real service, this would be read from input
        payload = {'client_id': 123}

        # Get a connection to the server ..
        conn = self.out.rest['REST Server'].conn

        # .. invoke it ..
        response = conn.get(self.cid, payload)

        # .. and handle the response here.
        ...

Calling HL7 FHIR

Similarly to REST endpoints, to invoke HL7 FHIR servers, fill out a form as below and let the "Security" field point to the OAuth definition just created. This will suffice for Zato to know when and how to use tokens received from the underlying OAuth server.

Here is sample code to invoke a FHIR server system - as with REST servers above, observe that we only refer to a connection by its name and Zato takes care of OAuth.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class GetPractitioner(Service):
    """ Returns a practictioner matching input data.
    """
    def handle(self) -> 'None':

        # Connection to use
        conn_name = 'My EHR'

        # In a real service, this would be read from input
        practitioner_id = 456

        # Get a connection to the server ..
        with self.out.hl7.fhir[conn_name].conn.client() as client:

            # Get a reference to a FHIR resource ..
            practitioners = client.resources('Practitioner')

            # .. look up the practitioner ..
            result = practitioners.search(active=True, _id=practitioner_id).get()

            # .. and handle the response here.
            ...

What about the API clients?

One aspect omitted above are the initial API clients - this is on purpose. How they invoke Zato, using what protocols, with what security mechanisms, and how to build responses based on their input data, this is completely independent of how Zato uses OAuth in its own communication with backend systems.

All of these aspects can and will be independent in practice, e.g. clients will use Basic Auth rather than OAuth. Or perhaps the clients will use AMQP, Odoo, SAP, or IBM MQ, without any HTTP, or maybe there will be no explicit API invocations and what we call "clients" will be actually CSV files in a shared directory that your services will be scheduled to periodically pick up. Yet, once more, regardless of what makes the input data available, the backend OAuth mechanism will work independently of it all.

Next steps

➤ API programming screenshots
➤ Python API integration tutorials
➤ More API programming examples in Python
➤ Open-source iPaaS in Python

HL7 FHIR API Integrations in Python

Mon, 16 Dec 2024 03:00:00 +0000

HL7 FHIR API Integrations in Python

2024-12-16, by Dariusz Suchojad

HL7 FHIR, pronounced "fire", is a data model and message transfer protocol designed to facilitate the exchange of information among systems used in health care settings.

In such environments, a FHIR server will assume the role of a central repository of health records with other systems integrating with it, potentially in a hub-and-spoke fashion, thus letting the FHIR server become a unified and consistent source of data that would otherwise stay confined to a silo of each individual health information system.

While FHIR is the way forward, the current reality of health care systems is that much of the useful and actionable information is distributed and scattered among many individual data sources - paper-based directories, applications or data bases belonging to the same or different enterprises - and that directly hampers the progress towards delivering good health care. Anyone witnessing health providers copy-and-pasting the same information from one application to another, not having access to the already existing data, not to mention people not having an easy way to access their own data about themselves either, can understand what the lack of interoperability looks like externally.

The challenges that integrators face are two-fold. On the one hand, the already existing systems, including software as well as medical appliances, were often not, or are still not being, designed for the contemporary inter-connected world. On the other hand, FHIR in itself is a relatively new technology which means that it is not straightforward to re-use the existing skills and competencies.

Zato is an open-source platform that makes it possible to integrate systems with FHIR using Python. Specifically, its support for FHIR enables quick on-boarding of integrators who may be new to health care interoperability, who are coming to FHIR with previous experience or interest in web development technologies, and who need an easy way to get started with and to navigate the complex landscape of health care integrations.

Connecting to FHIR servers

Outgoing FHIR connections are what allows Python-based services to communicate with FHIR servers. Throughout the rest of the chapter, the following definition will be used. It connects to a live, publicly available FHIR server.

Filling out the form below will suffice, there is no need for any server restarts. This principle, that restarts are not needed, applies all throughout the platform, whenever you change any piece of configuration, it will be automatically propagated as necessary.

Name: FHIR.Sample
Address: https://simplifier.net/zato
Security: No security definition (we will talk about security later)
TLS CA Certs: Default bundle

Retrieving data from FHIR servers

In Python code, you obtain client connections to FHIR servers through self.out.hl7.fhir objects, as in the example below which first refers to the server by its name and then looks up all the patients in the server.

The structure of the Patient resource that we expect to receive can be found here.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class FHIService1(Service):
    name = 'demo.fhir.1'

    def handle(self) -> 'None':

        # Connection to use
        conn_name = 'FHIR.Sample'

        with self.out.hl7.fhir[conn_name].conn.client() as client:

            # This is how we can refer to patients
            patients = client.resources('Patient')

            # Get all active patients, sorted by their birth date
            result = patients.sort('active', '-birthdate')

            # Log the result that we received
            for elem in result:
                self.logger.info('Received -> %s', elem['name'])

Invoking the service will store in logs the data expected:

INFO - Received -> [{'use': 'official', 'family': 'Chalmers', 'given': ['Peter', 'James']}]

For comparison, this is what the FHIR server displays in its frontend. - the information is the same.

Storing data in FHIR servers

To save information in a FHIR server, create the required resources and call .save to permanently store the data in the server. Resources can be saved either individually (as in the example below) or as a bundle.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class CommandsService(Service):
    name = 'demo.fhir.2'

    def handle(self) -> 'None':

        # Connection to use
        conn_name = 'FHIR.Sample'

        with self.out.hl7.fhir[conn_name].conn.client() as client:

            # First, create a new patient
            patient = client.resource('Patient')

            # Save the patient in the FHIR server
            patient.save()

            # Create a new appointment object
            appointment = client.resource('Appointment')

            # Who will attend it
            participant = {
                'actor': patient,
                'status':'accepted'
            }

            # Fill out the information about the appointment
            appointment.status = 'booked'
            appointment.participant = [participant]
            appointment.start  = '2022-11-11T11:11:11.111+00:00'
            appointment.end    = '2022-12-22T22:22:22.222+00:00'

            # Save the appointment in the FHIR server
            appointment.save()

Learning what FHIR resources to use

The "R" in FHIR stands for "Resources" and the sample code above uses resources such a Patient or Appointment but how does one learn what other resources exist and what they look like? In other words, how does one learn the underlying data model?

First, you need to get familiar with the spec itself which, in addition to textual information, offers visualizations of the data model. For instance, here is the description of the Observation object, including details such as all the attributes an Observation is composed of as well as their multiplicities.

Secondly, do spend time with FHIR servers such as Simplifier. Use Zato services to create test resources, look them up and compare the results with what the spec says. There is no substitute for experimentation when learning a new data model.

FHIR security

Outgoing FHIR connections can be secured in several ways, depending on what a given FHIR requires:

With Basic Auth definitions
With OAuth definitions
With SSL/TLS. If the server is not a public one (e.g. it is in a private network with a private IP address), you may need to upload the server's certificate to Zato first if you plan to use SSL/TLS because, without it, the server's certificate may be rejected.

MLLP, HL7 v2 and v3

While FHIR is what new deployments use, it is worth to add that there are still other HL7 versions frequently seen in integrations:

Version 2, using its own MLLP protocol
Version 3, using XML

Both of them can be used in Zato services, in both directions. For instance, it is possible to both receive HL7 v2 messages as well as to send them to external applications. It is also possible to send v2 messages using REST in addition to MLLP.

➤ Read more about using Python in API integrations
➤ Start the tutorial which will guide you how to design and build Python API services for interoperability, integrations and automation
➤ Open-source iPaaS in Python

New API integration tutorial in Python

Mon, 09 Dec 2024 03:00:00 +0000

New API integration tutorial in Python

2024-12-09, by Dariusz Suchojad

Do you know what airports, telecom operators, defense forces and health care organizations have in common?

They all rely heavily on deep-backend software systems which are integrated and automated using principled methodologies, innovative techniques and well-defined implementation frameworks.

If you'd like to learn how to integrate and automate such complex systems correctly, head over to the new API integration tutorial that will show you how to do it in Python too.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

# ##############################################################################

class MyService(Service):
    """ Returns user details by the person's name.
    """
    name = 'api.my-service'

    # I/O definition
    input = '-name'
    output = 'user_type', 'account_no', 'account_balance'

    def handle(self):

        # For later use
        name = self.request.input.name or 'partner'

        # REST connections
        crm_conn = self.out.rest['CRM'].conn
        billing_conn = self.out.rest['Billing'].conn

        # Prepare requests
        crm_request = {'UserName':name}
        billing_params = {'USER':name}

        # Get data from CRM
        crm_data = crm_conn.get(self.cid, crm_request).data

        # Get data from Billing
        billing_data = billing_conn.post(self.cid, params=billing_params).data

        # Extract the business information from both systems
        user_type = crm_data['UserType']
        account_no = crm_data['AccountNumber']
        account_balance = billing_data['ACC_BALANCE']

        self.logger.info(f'cid:{self.cid} Returning user details for {name}')

        # Now, produce the response for our caller
        self.response.payload = {
          'user_type': user_type,
          'account_no': account_no,
          'account_balance': account_balance,
        }

# ##############################################################################

Quick links

➤ API programming screenshots
➤ Here's the API integration tutorial again
➤ More API programming examples in Python
➤ Open-source iPaaS in Python

IMAP OAuth2 Microsoft 365 Integration

Tue, 19 Nov 2024 03:00:00 +0000

IMAP OAuth2 Microsoft 365 Integration

2024-11-19, by Dariusz Suchojad

Overview

This is the first in a series of articles about automation of and integrations with Microsoft 365 cloud products using Python and Zato.

We start off with IMAP automation by showing how to create a scheduled Python service that periodically pulls latest emails from Outlook using OAuth2-based connections.

IMAP and OAuth2

Microsoft 365 requires for all IMAP connections to use OAuth2. This can be challenging to configure in server-side automation and orchestration processes so Zato offers an easy way that lets you read and send emails without a need for getting into low-level OAuth2 details.

Consider a common orchestration scenario - a business partner sends automated emails with attachments that need to be parsed, some information needs to be extracted and processed accordingly.

Before OAuth2, an automation process would receive from Azure administrators a dedicated IMAP account with a username and password.

Now, however, in addition to creating an IMAP account, administrators will need to create and configure a few more resources that the orchestration service will use. Note that the password to the IMAP account will never be used.

Administrators need to:

Register an Azure client app representing your service that uses IMAP
Grant this app a couple of Microsoft Graph application permissions:
Mail.ReadWrite
Mail.Send

Next, administrators need to give you a few pieces of information about the app:

Application (client) ID
Tenant (directory) ID
Client secret

Additionally, you still need to receive the IMAP username (an e-mail address). It is just that you do not need its corresponding password.

In Dashboard

The first step is to create a new connection in your Zato Dashboard - this will establish an OAuth2-using connection that Zato will manage and your Python code will not have to do anything else, all the underlying OAuth2 tokens will keep refreshing as needed, the platform will take care of everything.

Having received the configuration details from Azure administrators, you can open your Zato Dashboard and navigate to IMAP connections:

Fill out the form as below, choosing "Microsoft 365" as the server type. The other type, "Generic IMAP" is used for the classical case of IMAP with a username and password:

Change the secret and click Ping to confirm that the connection is configured correctly:

In Python

Use the code below to receive emails. Note that it merely needs to refer to a connection definition by its name and there is no need for any usage of OAuth2 here:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class MyService(Service):

    def handle(self):

        # Connect to a Microsoft 365 IMAP connection by its name ..
        conn = self.email.imap.get('My Automation').conn

        # .. get all messages matching filter criteria ("unread" by default)..
        for msg_id, msg in conn.get():

            # .. and access each of them.
            self.logger.info(msg.data)

This is everything that is needed for integrations with IMAP using Microsoft 365 although we can still go further. For instance, to create a scheduled job to periodically invoke the service, go to the Scheduler job in Dashboard:

In this case, we decide to have a job that runs once per hour:

As expected, clicking OK will suffice for the job to start in background. It is as simple as that.

More resources

Web scraping API integrations

Wed, 13 Nov 2024 03:00:00 +0000

Web scraping API integrations

2024-11-13, by Dariusz Suchojad

Overview

In systems-to-systems integrations, there comes an inevitable time when we have to employ some kind of a web scraping tool to integrate with a particular application. Despite its not being our first choice, it is good to know what to use at such a time - in this article, I provide a gentle introduction to my favorite tool of this kind, called Playwright, followed by sample Python code that integrates it with an API service.

Naturally, in the context of backend integrations, web scraping should be avoided and, generally, it should be considered the last resort. The basic issue here is that while the UI term contains the "interface" part, it is not really the "Application Programming" Interface that we would like to have.

It is not that the UI cannot be programmed against. After all, a web browser does just that, it takes a web page and renders it as expected. Same goes for desktop or mobile applications. Also, anyone integrating with mainframe computers will recognize that this is basically what 3270 can be used for too.

Rather, the fundamental issue is that web scraping goes against the principles of separation of layers and roles across frontend, middleware and backend, which in turn means that authors of resources (e.g. HTML pages) do not really expect for many people to access them in automated ways.

Perhaps they actually should expect it, and web pages should finally start to resemble genuine knowledge graphs, easy to access by humans, be it manually or through automation tools, but the reality today is that it is not the case and, in comparison with backend systems, the whole of the web scraping space is relatively brittle, which is why we shun this approach in integrations.

Yet, another part of reality, particularly in enterprise integrations, is that people may be sometimes given access to a frontend application on an internal network and that is it. No API, no REST, no JSON, no POST data, no real data formats, and one is simply supposed to fill out forms as part of a business process.

Typically, such a situation will result in an integration gap. There will be fully automated parts in the business process preceding this gap, with multiple systems coordinated towards a specific goal and there will be subsequent steps in the process, also fully automated.

Or you may be given access only to a specific frontend and only through VPN via a single remote Windows desktop. Getting access to a REST API may take months or may be never realized because of some high level licensing issues. This is not uncommon in the real life.

Such a gap can be a jarring and sore point, truly ruining the whole, otherwise fluid, integration process. This creates a tension and to resolve the tension, we can, should all the attempts to find a real API fail, finally resort to web scraping.

It is mostly in this context that I am looking at Playwright below - the tool is good and it has many other uses that go beyond the scope of this text, and it is well worth knowing it, for instance for frontend testing of your backend systems, but, when we deal with API integrations, we should not overdo with web scraping.

Needless to say, if web scraping is what you do primarily, your perspective will be somewhat different - you will not need any explanation of why it is needed or when, and you may be only looking for a way to enclose up your web scraping code in API services. This article will explain that too.

Introducing Playwright

The nice part of Playwright is that we can use it to visually prepare a draft of Python code that will scrape a given resource. That is, instead of programming it in Python, we go to an address, fill out a form, click buttons and otherwise use everything as usually and Playwright generates for us code that will be later used in integrations.

That code will require a bit of clean-up work, which I will talk about below, but overall it works very nicely and is certainly useful. The result is not one of these do-not-touch auto-generated pieces of code that are better left to their own.

While there are better ways to integrate with Jira, I chose that application as an example of Playwright's usage simply because I cannot show you any internal application in a public blog post.

Below, there are two windows. One is Playwright's emulating a Blackberry device to open a resource. I was clicking around, I provided an email address and then I clicked the same email field once more. To the right, based on my actions, we can find the generated Python code, which I consider quite good and readable.

The Playwright Inspector, the tool that gave us the code, will keep recording all of our actions until we click the "Record" button which then allows us to click the button next to "Record" which is "Copy code to clipboard". We can then save the code to a separate file and run it on demand, automatically.

But first, we will need to install Playwright.

Installing and starting Playwright

The tools is written in TypeScript and can be installed using npx, which in turn is part of NodeJS.

Afterwards, the "playwright install" call is needed as well because that will potentially install runtime dependencies, such as Chrome libraries.

Finally, we install Playwright using pip as well because we want to access with Python. Note that if you are installing Playwright under Zato, the "/path/to/pip" will be typically "/opt/zato/code/bin/pip".

npx -g --yes playwright install
playwright install
/path/to/pip install playwright

We can now start it as below. I am using BlackBerry as an example of what Playwright is capable of. Also, it is usually more convenient to use a mobile version of a site when the main window and Inspector are opened side by side, but you may prefer to use Chrome, Firefox or anything else.

playwright codegen https://example.atlassian.net/jira --device "BlackBerry Z30"

That is practically everything as using Playwright to generate code in our context goes. Open the tool, fill out forms, copy code to a Python module, done.

What is still needed, though, is cleaning up the resulting code and embedding it in an API integration process.

Code clean-up

After you keep using Playwright for a while with longer forms and pages, you will note that the generated code tends to accumulate parts that repeat.

For instance, in the module below, which I already cleaned up, the same "[placeholder=\"Enter email\"]" reference to the email field is used twice, even if a programmer developing this could would prefer to introduce a variable for that.

There is not a good answer to the question of what to do about it. On the one hand, obviously, being programmers we would prefer not to repeat that kind of details. On the other hand, if we clean up the code too much, this may result in too much of a maintenance burden because we need to keep it mind that we do not really want to invest to much in web scraping and, should there be a need to repeat the whole process, we do not want to end up with Playwright's code auto-generated from scratch once more, without any of our clean-up.

A good compromise position is to at least extract any kind of credentials from the code to environment variables or a similar place and to remove some of the code comments that Playwright generates. The result as below is what it should like at the end. Not too much effort without leaving the whole code as it was originally either.

Save the code below as "play1.py" as this is what the API service below will use.

# -*- coding: utf-8 -*-

# stdlib
import os

# Playwright
from playwright.sync_api import Playwright, sync_playwright

class Config:
    Email = os.environ.get('APP_EMAIL', 'zato@example.com')
    Password = os.environ.get('APP_PASSWORD', '')
    Headless = bool(os.environ.get('APP_HEADLESS', False))

def run(playwright: Playwright) -> None:

    browser = playwright.chromium.launch(headless=Config.Headless) # type: ignore
    context = browser.new_context()

    # Open new page
    page = context.new_page()

    # Open project boards
    page.goto("https://example.atlassian.net/jira/software/projects/ABC/boards/1")
    page.goto("https://id.atlassian.com/login?continue=https%3A%2F%2Fexample.atlassian.net%2Flogin%3FredirectCount%3D1%26dest-url%3D%252Fjira%252Fsoftware%252Fprojects%252FABC%252Fboards%252F1%26application%3Djira")

    # Fill out the email
    page.locator("[placeholder=\"Enter email\"]").click()
    page.locator("[placeholder=\"Enter email\"]").fill(Config.Email)

    # Click #login-submit
    page.locator("#login-submit").click()

with sync_playwright() as playwright:
    run(playwright)

Web scraping as a standalone activity

We have the generated code so the first thing to do with it is to run it from command line. This will result in a new Chrome window's accessing Jira - it is Chrome, not Blackberry, because that is the default for Playwright.

The window will close soon enough but this is fine, that code only demonstrates a principle, it is not a full integration task.

python /path/to/play1.py

It is also useful that we can run the same Python module from our IDE, giving us the ability to step through the code line by line, observing what changes when and why.

Web scraping as an API service

Finally, we are ready to invoke the standalone module from an API service, as in the following code that we are also going to make available as a REST channel.

A couple of notes about the Python service below:

We invoke Playwright in a subprocess, as a shell command
We accept input through data models although we do not provide any output definition because it is not needed here
When we invoke Playwright, we set the APP_HEADLESS to True which will ensure that it does not attempt to actually display a Chrome window. After all, we intend for this service to run on Linux servers, in backend, and such a thing will be unlikely to work in this kind of an environment.

Other than that, this is a straightforward Zato service - it receives input, carries out its work and a reply is returned to the caller (here, empty).

# -*- coding: utf-8 -*-

# stdlib
from dataclasses import dataclass

# Zato
from zato.server.service import Model, Service

# ###########################################################################

@dataclass(init=False)
class WebScrapingDemoRequest(Model):
    email: str
    password: str

# ###########################################################################

class WebScrapingDemo(Service):
    name = 'demo.web-scraping'

    class SimpleIO:
        input  = WebScrapingDemoRequest

    def handle(self):

        # Path to a Python installation that Playwright was installed under
        py_path = '/path/to/python'

        # Path to a Playwright module with code to invoke
        playwright_path = '/path/to/the-playwright-module.py'

        # This is a template script that we will invoke in a subprocess
        command_template = """
        APP_EMAIL={app_email} APP_PASSWORD={app_password} APP_HEADLESS=True {py_path} {playwright_path}
        """

        # This is our input data
        input = self.request.input # type: WebScrapingDemoRequest

        # Extract credentials from the input ..
        email = input.email
        password = input.password

        # .. build the full command, taking all the config into account ..
        command = command_template.format(
            app_email = email,
            app_password = password,
            py_path = py_path,
            playwright_path = playwright_path,
        )

        # .. invoke the command in a subprocess  ..
        result = self.commands.invoke(command)

        # .. if it was not a success, log the details received ..
        if not result.is_ok:
            self.logger.info('Exit code -> %s', result.exit_code)
            self.logger.info('Stderr -> %s',    result.stderr)
            self.logger.info('Stdout -> %s',    result.stdout)

# ###########################################################################

Now, the REST channel:

The last thing to do is to invoke the service - I am using curl from the command line below but it could very well be Postman or a similar option.

curl localhost:17010/demo/web-scraping -d '{"email":"hello@example.com", "password":"abc"}' ; echo

There will be no Chrome window this time around because we run Playwright in the headless mode. There will be no output from curl either because we do not return anything from the service but in server logs we will find details such as below.

We can learn from the log that the command took close to 4 seconds to complete, that the exit code was 0 (indicating success) and that is no stdout or stderr at all.

INFO - Command `
APP_EMAIL=hello@example.com APP_PASSWORD=abc APP_HEADLESS=True
  /path/to/python
  /path/to/the-playwright-module.py
` completed in 0:00:03.844157,
  exit_code -> 0; len-out=0 (0 Bytes); len-err=0 (0 Bytes);
  cid -> zcmdc5422816b2c6ff9f10742134

We are now ready to continue to work on it - for instance, you will notice that the password is visible in logs and this should not be allowed.

But, all such works are extra in comparison with the main theme - we have Playwright, which is a a tool that allows us to quickly integrate with frontend applications and we can automate it through API services. Just as expected.

More resources

Meaningful automation in Python

Mon, 04 Nov 2024 03:00:00 +0000

Meaningful automation in Python

2024-11-04, by Dariusz Suchojad

This article is an introduction to meaningful automation, integrations and interoperability with Zato, service-oriented thinking and Python.

Zato is a convenient and secure, Python-based, open-source, service-oriented platform for automation, integrations and interoperability. It is used to connect distributed systems or data sources and to build API-focused, middleware and backend applications.

The platform is designed and built specifically with Python users in mind - often working in, and for, industries such as telecommunications, defense, health care and others that require automation, integrations and interoperability of multiple systems and processes.

Sample real-world, mission-critical Zato environments include:

Systems for telecommunication operators integrating CRM, ERP, Charging Systems, Billing and other OSS/BSS applications internal or external to the operators, including network automation of packet brokers and other network visibility and cybersecurity tools from Keysight
Enterprise services buses for government, helping in the digital transformation of legacy systems and processes towards modern capabilities
AI, ML and data science systems that analyze and improve acquisition and supply chain activities in government processes
Applied observability automation that enables meaningful decision making through the orchestration and coordination of the collection, distribution and presentation of data spread across a pool of independent systems
Platforms for health care and public administration systems, helping to achieve data interoperability through the integration of independent data sources, databases and health information exchanges (HIE)
Global IoT platforms for hybrid integrations of medical devices and software both in the cloud and on premises
Cybersecurity automation, including IT/OT hardware and software assets
Robotic process automation (RPA) of message flows and events produced by early warning systems

Zato offers connectors to all the popular technologies and vendors, such as REST, Cloud, task scheduling, Microsoft 365, Salesforce, Atlassian, SAP, Odoo, SQL, HL7, FHIR, AMQP, IBM MQ, LDAP, Redis, MongoDB, WebSockets, SOAP, Caching and many more.

Running in the cloud, on premises, or under Docker, Kubernetes and other container technologies, Zato services are optimized for high performance and security - it is easily possible to run hundreds and thousands of services on typical server instances as offered by Amazon, Google Cloud, Azure or other cloud providers.

Zato servers offer high availability and no-downtime deployment. Servers form clusters that are used to scale systems both horizontally and vertically.

The product is commercial open-source software with training, professional services and enterprise 24x7x365 support available.

A platform and language for interesting, reusable and atomic services

Zato promotes the design of, and helps you build, solutions composed of services that are interesting, reusable and atomic (IRA).

What does it really mean in practice that something is interesting, reusable and atomic? In particular, how do we define what is interesting?

Each interesting service should make its users want to keep using it more and more. People should immediately see the value of using the service in their processes. An interesting service strikes everyone as immediately useful in wider contexts, preferably with few or no conditions, prerequisites and obligations.

An interesting service is aesthetically pleasing, both in terms of its technical usage as well as its relevance to, and potential applicability in, fields broader than originally envisioned. If people check the service and say "I know, we will definitely use it" or "Why don't we use it" you know that the service is interesting. If they say "Oh no, not this one again" or "No, thanks, but no" then it is the opposite.

Note that focus here is on the value that the service brings for the user. You constantly need to keep in mind that people generally want to use services only if they allow them to fulfill their plans or execute some bigger ideas. Perhaps they already have them in mind and they are only looking for technical means of achieving that or perhaps it is your services that will make a person realize that something is possible at all, but the point is the same, your service should serve a grander purpose.

This mindset, of wanting to build things that are useful and interesting is not specific to Python or, indeed, to software and technology. Even if you are designing and implementing services for your own purposes, you need to act as if you were a consultant that can always see a bigger vision, a bigger architecture, and who can envision results that are still ahead in the future while at the same time not forgetting that it is always a series of small interesting actions, that everyone can relate to, that lead to success.

A curious observation can be made, particularly when you consider all the various aspects of the digital transformation that companies and organizations go through, is that many people to whom the services are addressed, or who sponsor their development, are surprised when they see what automation and integrations are capable.

Put differently, many people can only begin to visualize bigger designs once they see in practice smaller, practical results that further their missions, careers and otherwise help them at work. This is why, again, the focus on being interesting is essential.

At the same, it can be at times advantageous to you that people will not see automation or integrations coming. That lets you take the lead and build a center of such a fundamental shift around yourself. This is a great position to be in, a blue ocean of possibilities, because it means little to no competition inside an organization that you are a part of.

If you are your own audience, that is, if you build services for your own purposes, the same principles apply and it is easy to observe that thinking in services lets you build a toolbox of reusable, complementary capabilities, a portfolio, that you can take with you as you progress in your career. For instance, your services, and your work, can concentrate on a particular vendor and with a set of services that automate their products, you will be always able to put that into use, shorting your own development time, no matter who employs you and in what way.

Regardless of who the clients that you build the solutions for are, observe that automation and integrations with services are evolutionary and incremental in their nature, at least initially. Yes, the resulting value can often be revolutionary but you do not intend to incur any massive changes until there are clear, interesting results available. Trying to integrate and change existing systems at the same time is doable, but not trivial, and it is best left to later stages, once your automation gets the necessary buy-in from the organization.

Services should be ready to be used in different, independent processes. The processes can be strictly business ones, such as processing of orders or payments, or they can be of a deep, technical nature, e.g. automating cybersecurity hardware. What matters in either case is that reusability breeds both flexibility and stability.

There is inherent flexibility in being able to compose bigger processes out of smaller blocks with clearly defined boundaries, which can easily translate to increased competitive advantage when services are placed into more and more areas. A direct result of this is a reduction in R&D time as, over time, you are able to choose from a collection of loosely-coupled components, the services, that hide implementation details of a particular system or technology that they automate or integrate with.

Through their continued use in different processes, services can reduce overall implementation risks that are always part of any kind of software development - you will know that you can keep reusing stable functionality that has been already well tested and that is used elsewhere.

Because services are reusable, there is no need for gigantic, pure waterfall-style implementations of automation and integrations in an organization. Each individual project can contribute a smaller set of services that, as a whole, constitute the whole integrated environment. Conversely, each new project can start to reuse services delivered by the previous ones, hence allowing you to quickly, incrementally, prove the value of the investment in service-oriented thinking.

To make them reusable, services are designed in a way that hides their implementation details. Users only need to know how to invoke the service; the specific systems or processes it automates or integrates are not necessarily important for them to know as long as a specific business goal is achieved. Thanks to that, both services and what they integrate can be replaced without disrupting other parts - in reality, this is exactly what happens - systems with various kinds of data will be changed or modernized but the service will stay the same and the user will not notice anything.

Each service fulfills a single, atomic business need. Each service is deployed independently and, as a whole, they constitute an implementation of business processes taking place in your company or organization. Note that the definition of what the business need is, again, specific to your own needs. In purely market-oriented integrations, this may mean, for instance, the opening of a bank account. In IT or OT automation, on the other hand, it may mean the reconfiguration of a specific device.

That services are atomic also means that they are discrete and that their functionality is finely grained. You will recognize whether a design goes in this direction if consider the names of the services for a moment. An atomic service will invariably use a short name consisting of a single verb and noun. For instance, "Create Customer Account", "Stop Firewall", "Conduct Feasibility Study", it is easy to see that we cannot break them down into smaller part, they are atomic.

At the same time, you will keep creating composite services that invoke other services; this is natural and as expected but you will not consider services such as "Create Customer Account and Set Up a SIM Card" as atomic ones because, in that form, they will not be very reusable, and a major part of why being atomic is important is that it promotes reusability. For instance, having separate services to create customer accounts, independently of setting up their SIM cards, is that one can without difficulty foresee situations when an account is created but a SIM card is purchased at a later time and, conversely, surely one customer account should be able to potentially have multiple SIM cards. Think of it as being similar to LEGO bricks where just a few basic shapes can form millions of interesting combinations.

The point about service naming conventions is well worth remembering because this lets you maintain a vocabulary that is common to both technical and business people. A technical person will understand that such naming is akin to the CRUD convention from the web programming world while a business person will find it easy to map the meaning to a specific business function within a broader business process.

With Zato, you use Python to focus on the business logic exclusively and the platform takes care of scalability, availability communications protocols, messaging, security or routing. This lets you concentrate only on what is the very core of systems integrations - making sure their services are interesting, reusable and atomic.

Python is the perfect choice for this job because it hits the sweet spot under several key headings:

It is a very high level language, with a syntax close to how grammar of various spoken languages works, which makes it easy to translate business requirements into implementation
It is a solid, mainstream and full-featured, real programming language rather than a domain-specific one which means that it offers a great degree of flexibility and choice in expressing their needs
It is difficult to find universities without Python courses. Most people entering the workforce already know Python, it is a new career language. In fact, it is becoming more and more difficult to find new talent who would not prefer to use Python.
Yet, one does not need to be a developer or a full-time programmer to use Python. In fact, most people who use Python are not programmers at all. They are specialists in other fields who also need to use a programming language to automate or integrate their work in a meaningful way.
Many Python users come from backgrounds in network and cybersecurity engineering - fields that naturally require a lot of automation using a real language that is convenient and easy to get started with
Many Python users are scientists with a background in AI, ML and data science, applying their domain-specific knowledge in processes that, by their very nature, require them to collect and integrate data from independent sources, which again leads to automation and integrations
Many Python users have a strong web programming background which means that it takes little effort to take a step further, towards automation and integrations. In turn, this means that it is easy to find good people for API projects.
Many Python users know multiple programming languages - this is very useful in the context of integration projects where one is typically faced with dozens of technologies, vendors or integration methods and techniques.
Lower maintenance costs - thanks to the language's unique design, Python programmers tend to produce code that is easy to read and understand. From the perspective of multi-year maintenance, reading and analyzing code, rather than writing it, is what most programmers do most of the time, making sense to use a language that makes it easy to carry out the most common tasks.

In short, Python can be construed as executable pseudo-code with many of its users already having roots in modern server-side programming so Zato, both from a technical and strategic perspective, is a natural choice for both simple and complex, sophisticated automation, integration and interoperability solutions as a platform built in the language and designed for Python people from day one.

More resources

Salesforce API integrations and connected apps

Mon, 28 Oct 2024 03:00:00 +0000

Salesforce API integrations and connected apps

2024-10-28, by Dariusz Suchojad

Overview

This instalment in a series of articles about API integrations with Salesforce covers connected apps - how to create them and how to obtain their credentials needed to exchange REST messages with Salesforce.

In Salesforce's terminology, a connected app is, essentially, an API client. It has credentials, a set of permissions, and it works on behalf of a user in an automated manner.

In particular, the kind of a connected app that I am going to create below is one that can be used in backend, server-side integrations that operate without any direct input from end users or administrators, i.e. the app is created once, its permissions and credentials are set once, and then it is able to work uninterrupted in the background, on server side.

Server-side systems are quite unlike other kinds of apps, such as mobile ones, that assume there is a human operator involved - they have their own work characteristics, related yet different, and I am not going to cover them here.

Note that permission types and their scopes are a separate, broad subject and they will described in a separate how-to article.

Finally, I assume that you are either an administrator in a Salesforce organization or that you are preparing information for another person with similar grants in Salesforce.

Conceptually, there is nothing particularly unusual about Salesforce connected apps, it is just its own mini-world of jargon and, at the end of the day, it simply enables you to invoke APIs that Salesforce is built on. It is just that knowing where to click, what to choose and how to navigate the user interface can be a daunting challenge that this article hopes to make easier to overcome.

The steps

For an automated, server-side connected app to make use of Salesforce APIs, the requirements are:

Having access to username/password credentials
Creating a connected app
Granting permissions to the app (not covered in this article)
Obtaining a customer key and customer secret for the app

You will note that there are four credentials in total:

Username
Password
Customer key
Customer secret

Also, depending on what chapter of the Salesforce documentation you are reading, you will note that the customer key can be also known as "client_id" whereas another name for the customer secret is "client_secret". These two pairs mean the same.

Access to username/password credentials

For starters, you need to have an account in Salesforce, a combination of username + password that you can log in with and on whose behalf the connected app will be created:

Creating a connected app

Once you are logged in, go to Setup in the top right-hand corner:

In the search box, look up "app manager":

Next, click the "New Connected App" button to the right:

Fill out the basic details such as "Connect App Name" and make sure that you select "Enable OAuth Settings". Then, given that in this document we are not dealing with the subject of permissions at all, grant full access to the connected app and finally click "Save" at the bottom of the page.

Obtaining a customer key and customer secret

We have a connected app but we still do not know what its customer key and secret are. To reveal it, go to the "App Manager" once more, either via the search box or using the menu on the left hand side.

Find your app in the list and click "View" in the list of actions. Observe that it is "View", not "Edit" or "Manage", where you can check what the credentials are:

The customer key and secret van be now revealed in the "API (Enable OAuth Settings)" section:

This concludes the process - you have a connected app and all the credentials needed now.

Testing

Seeing as this document is part of a series of how-tos in the context of Zato, if you would like to integrate with Salesforce in Python, at this point you will be able to follow the steps in another where everything is detailed separately.

Just as a quick teaser, it would look akin to the below.

...

# Salesforce REST API endpoint to invoke
path = '/sobjects/Campaign/'

# Build the request to Salesforce based on what we received
request = {
  'Name': input.name,
  'Segment__c': input.segment,
}

# Create a reference to our connection definition ..
salesforce = self.cloud.salesforce['My Salesforce Connection']

# .. obtain a client to Salesforce ..
with salesforce.conn.client() as client: # type: SalesforceClient

    # .. create the campaign now.
    response = client.post(path, request)

...

On a much lower level, however, if you would just like to quickly test out whether you configured the connected app correctly, you can invoke from command line a Salesforce REST endpoint that will return an OAuth token, as below.

Note that, as I mentioned it previously, client_id is the same as customer key and client_secret is the same as customer secret.

curl https://example.my.salesforce.com/services/oauth2/token \
   -H "X-PrettyPrint: 1" \
  --header 'Content-Type: application/x-www-form-urlencoded' \
  --data-urlencode 'grant_type=password' \
  --data-urlencode 'username=hello@example.com' \
  --data-urlencode 'password=my.password' \
  --data-urlencode 'client_id=my.customer.key' \
  --data-urlencode 'client_secret=my.client.secret'

The result will be, for instance:

{
  "access_token" : "008e0000000PTzLPb!4Vzm91PeIWJo.IbPzoEZf2ygEM.6cavCt0YwAGSM",
  "instance_url" : "https://example.my.salesforce.com",
  "id" : "https://login.salesforce.com/id/008e0000000PTzLPb/0081fSUkuxPDrir000j1",
  "token_type" : "Bearer",
  "issued_at" : "1649064143961",
  "signature" : "dwb6rwNIzl76kZq8lQswsTyjW2uwvTnh="
}

Above, we have an OAuth bearer token on output - this can be used in subsequent, business REST calls to Salesforce but how to do it exactly in practice is left for another article.

More resources

HL7 FHIR Security with Basic Auth, OAuth and SSL/TLS

Mon, 21 Oct 2024 03:00:00 +0000

HL7 FHIR Security with Basic Auth, OAuth and SSL/TLS

2024-10-21, by Dariusz Suchojad

HL7 FHIR Security

Preliminary reading: HL7 FHIR Integrations in Python

FHIR servers offer their APIs using REST, which in turn means that they are HTTP servers under the hood. As a result, a few common security mechanisms can be employed to secure access to the servers when you invoke them.

Basic Auth
OAuth
SSL/TLS encryption

When you integrate with a FHIR server, consult with its maintainers what of these, if any, should be used.

Note that Basic Auth and OAuth, when used over HTTP, are mutually exclusive. The HTTP protocol reserves only one header, called Authorization, for the authorization credentials and there is no standard way in the protocol to use more than one authorization mechanism.

On the other hand, the SSL/TLS encryption is independent of the credentials used and can be employed with Basic Auth, OAuth, or in scenarios when no authorization header is sent at all.

This is why, when you create or update a FHIR connection, you can set Basic Auth or OAuth and SSL/TLS options independently, as in the screenshot below:

Basic Auth

Basic Auth is a combination of username and password credentials. They are allocated in advance by the maintainers of a FHIR server.

To use them in your integrations, create a new Basic Auth definition in the Dashboard, set its password, and assign it to an existing or new outgoing FHIR connection. No restarts nor reloads are required.

OAuth

Authentication with OAuth is built around a notion of short-lived tokens, which are simple strings. Your Zato server has credentials, much like a username and password although they are called a client ID and secret, based on which an authentication server issues tokens that let Zato prove that in fact it does have the correct credentials and that it is allowed to invoke a particular API as it is defined through scopes.

Scopes can be likened to permissions to access a subset of resources from a FHIR server. When Zato receives a token from an authentication server the token is inherently linked to specific scopes within which it can interact with the FHIR server.

That tokens are short-lived means that they need to be refreshed periodically, e.g. at least once per hour Zato needs to ask the authentication server for a new token based on its credentials. This process takes place under the hood and requires no configuration on your part.

To use OAuth in your integrations, create a new OAuth definition in the Dashboard, set its secret, and assign it to an existing or new outgoing FHIR connection. No restarts nor reloads are required.

SSL/TLS encryption

If the FHIR server that a connection points to uses SSL/TLS then a question arises of how to validate the certificate of the Certificate Authority (CA) that signed the certificate that the FHIR server uses. There are several options:

Skip validation - the certificate will not be validated at all and it will be always accepted.
Default bundle - the certificate will have to belong to one of the publicly recognized CAs. The bundle contains the same certificates that common web browsers do so this option is good if the FHIR server is available in the public Internet, e.g. https://fhir.example.com
Custom bundle - if the FHIR server's certificate was signed by an internal, non-public CA then the CA's certificate bundle, in the PEM format, can be uploaded through the Dashboard to make it possible to choose it when the FHIR connection is being created or updated.

More resources

API testing in pure English

Mon, 07 Oct 2024 03:00:00 +0000

API testing in pure English

2024-10-07, by Dariusz Suchojad

How to test APIs in pure English

Do you have 20 minutes to learn how to test APIs in pure English, without any programming needed?

Great, the API testing tutorial is here.

Right after you complete it, you'll be able to write API tests as the one below.

Next steps:

➤ Read about how to use Python to build and integrate enterprise APIs that your tests will cover
➤ Python API integration tutorials
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?
➤ Open-source iPaaS in Python

Akenza IoT Python WebSockets

Mon, 16 Sep 2024 03:00:00 +0000

Akenza IoT Python WebSockets

2024-09-16, by Dariusz Suchojad

Overview

The Akenza IoT platform, on its own, excels in collecting and managing data from a myriad of IoT devices. However, it is integrations with other systems, such as enterprise resource planning (ERP), customer relationship management (CRM) platforms, workflow management or environmental monitoring tools that enable a complete view of the entire organizational landscape.

Complementing Akenza's capabilities, and enabling the smooth integrations, is the versatility of Python programming. Given how flexible Python is, the language is a natural choice when looking for a bridge between Akenza and the unique requirements of an organization looking to connect its intelligent infrastructure.

This article is about combining the two, Akenza and Python. At the end of it, you will have:

A bi-directional connection to Akenza using Python and WebSockets
A Python service subscribed to and receiving events from IoT devices through Akenza
A Python service that will be sending data to IoT devices through Akenza

Since WebSocket connections are persistent, their usage enhances the responsiveness of IoT applications which in turn helps to exchange occurs in real-time, thus fostering a dynamic and agile integrated ecosystem.

Python and Akenza WebSocket connections

First, let's have a look at full Python code - to be discussed later.

 # -*- coding: utf-8 -*-

# Zato
from zato.server.service import WSXAdapter

# ###############################################################################################
# ###############################################################################################

if 0:
    from zato.server.generic.api.outconn.wsx.common import OnClosed, \
        OnConnected, OnMessageReceived

# ###############################################################################################
# ###############################################################################################

class DemoAkenza(WSXAdapter):

    # Our name
    name = 'demo.akenza'

    def on_connected(self, ctx:'OnConnected') -> 'None':
        self.logger.info('Akenza OnConnected -> %s', ctx)

# ###############################################################################################

    def on_message_received(self, ctx:'OnMessageReceived') -> 'None':

        # Confirm what we received
        self.logger.info('Akenza OnMessageReceived -> %s', ctx.data)

        # This is an indication that we are connected ..
        if ctx.data['type'] == 'connected':

            # .. for testing purposes, use a fixed asset ID ..
            asset_id:'str' = 'abc123'

            # .. build our subscription message ..
            data = {'type': 'subscribe', 'subscriptions': [{'assetId': asset_id, 'topic': '*'}]}

            ctx.conn.send(data)

        else:
            # .. if we are here, it means that we received a message other than type "connected".
            self.logger.info('Akenza message (other than "connected") -> %s', ctx.data)

# ##############################################################################################

    def on_closed(self, ctx:'OnClosed') -> 'None':
        self.logger.info('Akenza OnClosed -> %s', ctx)

# ##############################################################################################
# ##############################################################################################

Now, deploy the code to Zato and create a new outgoing WebSocket connection. Replace the API key with your own and make sure to set the data format to JSON.

Receiving messages from WebSockets

The WebSocket Python services that you author have three methods of interest, each reacting to specific events:

on_connected - Invoked as soon as a WebSocket connection has been opened. Note that this is a low-level event and, in the case of Akenza, it does not mean yet that you are able to send or receive messages from it.
on_message_received - The main method that you will be spending most time with. Invoked each time a remote WebSocket sends, or pushes, an event to your service. With Akenza, this method will be invoked each time Akenza has something to inform you about, e.g. that you subscribed to messages, that
on_closed - Invoked when a WebSocket has been closed. It is no longer possible to use a WebSocket once it has been closed.

Let's focus on on_message_received, which is where the majority of action takes place. It receives a single parameter of type OnMessageReceived which describes the context of the received message. That is, it is in the "ctx" that you will both the current request as well as a handle to the WebSocket connection through which you can reply to the message.

The two important attributes of the context object are:

ctx.data - A dictionary of data that Akenza sent to you
ctx.conn - The underlying WebSocket connection through which the data was sent and through you can send a response

Now, the logic from lines 30-40 is clear:

First, we check if Akenza confirmed that we are connected (type=='connected'). You need to check the type of a message each time Akenza sends something to you and react to it accordingly.
Next, because we know that we are already connected (e.g. our API key was valid) we can subscribe to events from a given IoT asset. For testing purposes, the asset ID is given directly in the source code but, in practice, this information would be read from a configuration file or database.
Finally, for messages of any other type we simply log their details. Naturally, a full integration would handle them per what is required in given circumstances, e.g. by transforming and pushing them to other applications or management systems.

A sample message from Akenza will look like this:

INFO - WebSocketClient -  Akenza message (other than "connected") -> {'type': 'subscribed',
'replyTo': None, 'timeStamp': '2023-11-20T13:32:50.028Z',
'subscriptions': [{'assetId': 'abc123', 'topic': '*', 'tagId': None, 'valid': True}],
'message': None}

How to send messages to WebSockets

An aspect not to be overlooked is communication in the other direction, that is, sending of messages to WebSockets. For instance, you may have services invoked through REST APIs, or perhaps from a scheduler, and their job will be to transform such calls into configuration commands for IoT devices.

Here is the core part of such a service, reusing the same Akenza WebSocket connection:

 # -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

# ##############################################################################################
# ##############################################################################################

class DemoAkenzaSend(Service):

    # Our name
    name = 'demo.akenza.send'

    def handle(self) -> 'None':

        # The connection to use
        conn_name = 'Akenza'

        # Get a connection ..
        with self.out.wsx[conn_name].conn.client() as client:

            # .. and send data through it.
            client.send('Hello')

# ##############################################################################################
# ##############################################################################################

Note that responses to the messages sent to Akenza will be received using your first service's on_message_received method - WebSockets-based messaging is inherently asynchronous and the channels are independent.

Now, we have a complete picture of real-time, IoT connectivity with Akenza and WebSockets. We are able to establish persistent, responsive connections to assets, we can subscribe to and send messages to devices, and that lets us build intelligent automation and integration architectures that make use of powerful, emerging technologies.

More resources

Automating telecommunications networks with Python and SFTP

Mon, 29 Jul 2024 03:00:00 +0000

Automating telecommunications networks with Python and SFTP

2024-07-29, by Dariusz Suchojad

In telecommunications, the Secure File Transfer Protocol (SFTP) serves as a critical mechanism for secure and reliable file exchange between different network components devices, and systems, whether it is updating configurations, network monitoring, exchanging customer data, or facilitating software updates. Conversely, Python is an ideal tool for the automation of telecommunications networks thanks to its readability and versatility.

Let's dive into how to employ the two effectively and efficiently using the Zato integration and automation platform.

Dashboard

The first step is to define a new SFTP connection in your Dashboard, as in the screenshots below.

The form lets you provide all the default options that apply to each SFTP connection - remote host, what protocol to use, whether file metadata should be preserved during transfer, logging level and other details that you would typically provide.

Simply fill it out with the same details that you would use if it were command line-based SFTP connections.

Pinging

The next thing, right after the creation of a new connection, is to ping it to check if the server is responding.

Pinging opens a new SFTP connection and runs the ping command - in the screenshot above it was ls . - a practically no-op command whose sole purpose is to let the connection confirm that commands in fact can be executed, which proves the correctness of the configuration.

This will either returns details of why a connection could not be established or the response time if it was successful.

Cloud SFTP console

Having validated the configuration by pinging it, we can now execute SFTP commands straight in Dashboard from a command console:

Any SFTP command, or even a series of commands, can be sent and responses retrieved immediately. It is also possible to increase the logging level for additional SFTP protocol-level details.

This makes it possible to rapidly prototype file transfer functionality as a series of scripts that can be next moved as they are to Python-based services.

Python automation

Now, in Python, your API automation services have access to an extensive array of capabilities - from executing transfer commands individually or in batches to the usage of SFTP scripts previously created in your Dashboard.

Here is how Python can be used in practice:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class MySFTPService(Service):
    def handle(self):

        # Connection to use
        conn_name = 'My SFTP Connection'

        # Get a handle to the connection object
        conn = self.out.sftp[conn_name].conn

        # Execute an arbitrary script with one or more SFTP commands, like in web-admin
        my_script = 'ls -la /remote/path'
        conn.execute(my_script)

        # Ping a remote server to check if it responds
        conn.ping()

        # Download an entry, possibly recursively
        conn.download('/remote/path', '/local/path')

        # Like .download but remote path must point to a file (exception otherwise)
        conn.download_file('/remote/path', '/local/path')

        # Makes the contents of a remote file available on output
        out = conn.read('/remote/path')

        # Uploads a local file or directory to remote path
        conn.upload('/local/path', '/remote/path')

        # Writes input data out to a remote file
        data = 'My data'
        conn.write(data, '/remote/path')

        # Create a new directory
        conn.create_directory('/path/to/new/directory')

        # Create a new symlink
        conn.create_symlink('/path/to/new/symlink')

        # Create a new hard-link
        conn.create_hardlink('/path/to/new/hardlink')

        # Delete an entry, possibly recursively, no matter what kind it is
        conn.delete('/path/to/delete')

        # Like .delete but path must be a directory
        conn.delete_directory('/path/to/delete')

        # Like .delete but path must be a file
        conn.delete_file('/path/to/delete')

        # Like .delete but path must be a symlink
        conn.delete_symlink('/path/to/delete')

        # Get information about an entry, e.g. modification time, owner, size and more
        info = conn.get_info('/remote/path')

        self.logger.info(info.last_modified)
        self.logger.info(info.owner)
        self.logger.info(info.size)
        self.logger.info(info.size_human)
        self.logger.info(info.permissions_oct)

        # A boolean flag indicating if path is a directory
        result = conn.is_directory('/remote/path')

        # A boolean flag indicating if path is a file
        result = conn.is_file('/remote/path')

        # A boolean flag indicating if path is a symlink
        result = conn.is_symlink('/remote/path')

        # List contents of a directory - items are in the same format that .get_info uses
        items = conn.list('/remote/path')

        # Move (rename) remote files or directories
        conn.move('/from/path', '/to/path')

        # An alias to .move
        conn.rename('/from/path', '/to/path')

        # Change mode of entry at path
        conn.chmod('600', '/path/to/entry')

        # Change owner of entry at path
        conn.chown('myuser', '/path/to/entry')

        # Change group of entry at path
        conn.chgrp('mygroup', '/path/to/entry')

Summary

Given how important SFTP is in telecommunications, having a convenient and easy way to automate it using Python is an essential ability in a network engineer's skill-set.

Thanks to the SFTP connections in Zato, you can prototype SFTP scripts in Dashboard and employ them in API services right after that. To complement it, a full Python API is available for programmatic access to remote file servers.

Combined, the features make it possible to create scalable and reusable file transfer services in a quick and efficient manner using the most convenient programming language, Python.

More resources

➤ Click here to read more about using Python and Zato in telecommunications
➤ What is a Network Packet Broker? How to automate networks in Python?
➤ What is an integration platform?
➤ Python Integration platform as a Service (iPaaS)
➤ What is an Enterprise Service Bus (ESB)? What is SOA?
➤ Open-source iPaaS in Python

Integrating with WordPress and Elementor API webhooks

Fri, 21 Apr 2023 03:00:00 +0000

Integrating with WordPress and Elementor API webhooks

2023-04-21, by Dariusz Suchojad

Overview

Consider this scenario:

You have a WordPress instance, possibly installed in your own internal network
With WordPress, you use Elementor, a popular website builder
A user fills out a form that you prepared using Elementor, e.g. the user provides his or her email and username to create an account in your CRM
Now, after WordPress processes this information accordingly, you also need to send it all to a remote backend system that only accepts JSON messages

The concern here is that WordPress alone will not send it to the backend system.

Hence, we are going to use an Elementor-based webhook that will invoke Zato which will be acting as an integration layer. In Zato, we will use Python to transform the results of what was submitted in the form - in order to deliver it to the backend API system using a REST call.

Creating a channel

A Zato channel is a way to describe the configuration of a particular API endpoint. In this case, to accept data from WordPress, we are going to use REST channels:

In the screenshot below, note particularly the highlighted data format field. Typically, REST channels will use JSON, but here, we need to use "Form data" because this is what we are getting from Elementor.

Now, we can add the actual code to accept the data and to communicate with the remote, backend system.

Python code

Here is the Python code and what follows is an explanation of how it works:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

# Field configuration
field_email = 'fields[email][value]'
field_username = 'fields[username][value]'

class CreateAccount(Service):

    # The input that we expect from WordPress, i.e. what fields it needs to send
    input = field_email, field_username

    def handle(self):

        # This is a dictionary object with data from WordPress ..
        input = self.request.input

        # .. so we can use dictionary access to extract values received ..
        email = input[field_email]
        username = input[field_username]

        # .. now, we can create a JSON request for the backend system ..
        # .. again, using a regular Python dictionary ..
        api_request = {
            'Email': email
            'Username': username
        }

        # Obtain a connection to the backend system ..
        conn = self.out.rest['CRM'].conn

        # .. invoke that system ..
        # Invoke the resource providing all the information on input
        response = conn.post(self.cid, api_request)

        # .. and log the response received ..
        self.logger.info('Backend response -> %s', response.data)

The format of data that Elementor will use is of a specific nature. It is not JSON and the field names are not sent directly either.

That is, if your form has fields such as "email" and "username", what the webhook sends is named differently. The names will be, respectively:

fields[email][value]
fields[username][value]

If it were JSON, we could say that instead of this ..

{
  "email": "hello@example.com",
  "username": "hello"
}

.. the webhook was sending to you that:

{
  "fields[email][value]": "hello@example.com",
  "fields[username][value]": "hello"
}

The above format explains why in the Python code below we are extracting all the input fields from WordPress using the "self.request.input" object using its dictionary access syntax method.

Normally, if the field was plain "username", we would be doing "self.request.input.username" but this is not available in this case because of the naming conventions of the fields from Elementor.

Now, the only remaining part is the definition of the outgoing REST connection that service should use.

Outgoing REST connections

Create an outgoing REST connection as below - this time around, note that the data format is JSON.

Using the REST channel

In your WordPress dashboard, create a webhook using Elementor and point it to the channel created earlier, e.g. make Elementor invoke an address such as http://10.157.11.39:11223/api/wordpress/create-account

Each time a form is submitted, its contents will go to Zato, your service will transform it to JSON and the backend CRM system will be invoked.

And this is everything - you have just integrated WordPress, Elementor webhooks and an external API backend system in Python.

More resources

How to invoke REST APIs from Zato services

Tue, 23 Aug 2022 03:00:00 +0000

How to invoke REST APIs from Zato services

2022-08-23, by Dariusz Suchojad

This Zato article is a companion to an earlier post - previously, we covered accepting REST API calls and now we will look at how Zato services can invoke external REST endpoints.

Outgoing connections

Similar to how channels are responsible for granting access to your services via REST or other communication means, it is outgoing connections (outconns, as an abbreviation) that let the services access resources external to Zato, including REST APIs.

Here is a sample definition of a REST outgoing connection:

The Python implementation will follow soon but, for now, let's observe that keeping the two separate has a couple of prominent advantages:

The same outgoing connection can be used by multiple services
The configuration is maintained in one place only - any change is immediately reflected on all servers and services can make use of the new configuration without any interruptions

Most of the options of REST outconns have the same meaning as with channels but TLS CA certs may require particular attention. This option dictates what happens if a REST endpoint is invoked using HTTPS rather than HTTP, how the remote end's TLS certificate is checked.

The option can be one of:

Default bundle - a built-in bundle of CA certificates will be used for validation. This is the same bundle that Mozilla uses and is a good choice if the API you are invoking is a well-known, public one with endpoints signed by one of the public certificate authorities.
If you upload your own CA certificates, they can be used for validation of external REST APIs - for instance, your company or a business partner may have their own internal CAs
Skip validation - no validation will be performed at all, any TLS certificate will be accepted, including self-signed ones. Usually, this option should not be used for non-development purposes.

Python code

A sample service making use of the outgoing connection is below.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class GetUserDetails(Service):
    """ Returns details of a user by the person's name.
    """
    name = 'api.user.get-details'

    def handle(self):

        # In practice, this would be an input parameter
        user_name = 'john.doe'

        # Name of the connection to use
        conn_name = 'My REST Endpoint'

        # Get a connection object
        conn = self.out.rest[conn_name].conn

        # A single dictionary with all the parameters,
        # Zato will know where each one goes to.
        params = {
            'name': user_name,
            'app_id': 'Zato',
            'app_version': '4.1'
        }

        # We are responsible for catching exceptions
        try:

            # Invoke the endpoint
            response = conn.get(self.cid, params)

            # Log the response
            self.logger.info('Response `%s`', response.data)

        # We caught an exception - log its details
        except Exception as e:
            self.logger.warn('Endpoint could not be invoked due to `%s`', e.args[0])

First, we obtain a connection handle, then the endpoint is invoked and a response is processed, which in this case means merely outputting its contents to server logs.

In the example, we use a constant user name and query string but in practice, they would be likely produced basing on user input.

Note the 'params' dictionary - when you invoke this service, Zato will know that 'name' should go to the URL path but all the remaining parameters will go to the request's query string. Again, this gives you additional flexibility, e.g. if the endpoint's URL changes from path parameters to query string, the service will continue to work without any changes.

Observe, too, that we are responsible for catching and handling potential exceptions arising from invoking REST endpoints.

Finally - because the outgoing connection's data format is JSON, you are not required to de-/serialize it yourself, the contents of 'response.data' is already a Python dict read from a JSON response.

At this point, the service is ready to be invoked - let's say, through REST, AMQP or from the scheduler - and when you do it, here is the output that will be seen in server logs:

INFO - Response `{'user_id': 123, 'username': 'john.doe', 'display_name': 'John Doe'}`

Now, you can extend it to invoke other systems, get data from an SQL database or integrate with other APIs.

How to integrate with Confluence APIs

Tue, 21 Jun 2022 03:00:00 +0000

How to integrate with Confluence APIs

2022-06-21, by Dariusz Suchojad

In a previous article, I talked about Jira, and if you are a Jira user, chances are that you also use Confluence as they often go hand in hand, Jira as a ticketing application and Confluence as an enterprise knowledge management system.

From the perspective of integrations, connecting to Confluence and invoking its APIs looks and feels practically the same as with Jira:

You need an API token
You fill out a form in Zato Dashboard
You create a Python service that offers methods such as get_page_by_title, attach_file, update_page_property and similar.

Let's go through it all step-by-step, starting off with the creation of an API token.

Creating an Atlassian API token

To invoke Confluence, you use an API token that can be shared with other Atlassian products, such as Jira. If you do not have one already, here is how to create it:

Log in to Confluence or Jira
Visit the address where API tokens can be managed: https://id.atlassian.com/manage-profile/security/api-tokens
Click "Create API Token" and provide a name for the token, such as "Zato Integrations"
Copy the token somewhere as, once it has been created, you will not be able to retrieve it later on. The only way to change a token is to revoke it and create a new one.

Creating a Confluence connection

In your Zato Dashboard, go to Cloud -> Atlassian -> Confluence:

Click "Create a new connection" and fill out the form below. The username is the same as the email address that you log in to Confluence or Jira with.

Now, click "Change API Token" and enter the token created in the previous section:

Invoking Confluence

Authoring a Zato service that invokes Confluence follows a pattern that will feel familiar no matter what kind of an API you integrate with:

Obtain a connection to remote resource
Invoke it
Process the response the resource returned

In the case of the code below, we are merely logging the response from Confluence. In a bigger integration, we would process it accordingly, e.g. parts of the output could be synchronized with Jira or another system.

Note the 'client.get_all_pages_from_space' method below - the client will offer other methods as well, e.g. get_space, get_page_as_pdf or ways to run CQL (Confluence Query Language) directly. Use auto-completion in your IDE to discover all the methods available.

# -*- coding: utf-8 -*-

# Zato
from zato.common.typing_ import cast_
from zato.server.service import Service

# ###########################################################################

if 0:
    from zato.server.connection.confluence_ import ConfluenceClient

# ###########################################################################

class GetAllPages(Service):

    def handle(self):

        # Name of the Confluence space that our pages are in
        space = 'ABC'

        # Name of the connection definition to use
        conn_name = 'My Confluence Connection'

        # .. create a reference to our connection definition ..
        confluence = self.cloud.confluence[conn_name]

        # .. obtain a client to Confluence ..
        with confluence.conn.client() as client: # type: ConfluenceClient

            # Cast to enable code completion
            client = cast_('ConfluenceClient', client)

            # Get all pages from our space
            pages = client.get_all_pages_from_space(space)

            self.logger.info('Pages received -> %s', pages)

# ###########################################################################

That is all - you have create an Atlassian API token, a Zato Confluence connection and you have integrated with Confluence in Python!

Enterprise API integrations under Windows

Wed, 15 Sep 2021 03:00:00 +0000

Enterprise API integrations under Windows

2021-09-15, by Dariusz Suchojad

If you are on Windows, looking for an integration platform to connect your APIs, systems, applications, backend resources or mobile apps, I am happy to let you know that Zato now supports Windows natively in addition to Linux.

You can use the platform to connect your systems, to offer new APIs on top of existing server-side resources, to secure your applications, to connect IoT devices, to move and distribute static files or to grant backend access to online users - these are just a few examples of how Zato is typically used. Read on for more details on how to install it under Windows.

Installing Zato

The native Windows port is new in Zato 4.1 - download an .msi package for your Python version and install it by following the steps here.

The end result will be a 'zato' command line tool, as below:

The first integration cluster

Run the command below to create your first cluster. It is called "qs-1" because this is the first quickstart cluster in this system:

Note the indicated user name (admin) and the password - after the cluster is started, you will be able to use the credentials to log in to your Dashboard in a moment.

Let's start the cluster in this case - it will open a new PowerShell or cmd.exe window with all the logs in foreground.

PS C:\> C:\Users\Administrator\env\qs-1\zato-qs-start.bat

Now, you can log in to Dashboard at http://localhost:8183 to look around and familiarize yourself with the GUI.

That is practically everything, the cluster is up and running, there is a server, Dashboard and a background scheduler. You are ready to begin the tutorial now.

Diving deep into REST API channels

Mon, 04 Jan 2021 03:00:00 +0000

Diving deep into REST API channels

2021-01-04, by Dariusz Suchojad

We begin in 2021 with a deep dive into Zato REST API channels. What are they? How to use them efficiently? How can they configured for maximum flexibility? Read on to learn all the details.

A sample service

First, let's have a look at a sample service that we want to make available to API clients.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class GetUserDetails(Service):
    """ Returns details of a selected user.
    """
    name = 'api.user.get-details'

    class SimpleIO:
        input_required = 'user_name'
        output_required = 'email', 'user_type'

    def handle(self):

        # Log what we are about to do.
        self.logger.info('Returning details of `%s`', self.request.input.user_name)

        # In real code, we would look up the details in a database,
        # but not necessarily in a cache - read the article why it is not needed.
        details = {
            'email': 'my.user@example.com',
            'user_type': 'ABC'
        }

        # Return the response now.
        self.response.payload = details

The first thing that may strike you is that the code is on a very high level - it just has access to a user_name and some data is returned but there is no mention of REST, no data serialization, caching or anything that is not the business functionality of returning user details.

This is by design. In Zato, services focus on what they actually need to do and the lower-level details are left to the platform. In other words, the service does not need to be concerned with peculiarities of a given transport method, it just has its input to process and output to produce. It is channels in front of a service that deal with all such aspects and the service concentrates on higher level logic.

This makes it possible to employ the same service in other contexts - for instance, we are describing REST channels today but the very same service could be invoked from the scheduler, through AMQP, IBM MQ or via other channels, including multiple REST ones, helping you in this way design a reusable Service-Oriented Architecture (SOA).

With that in mind, we can advance to REST channels now.

REST API channels

In Zato web-admin, a definition of a sample REST channel making use of our service may look like below.

Now that we have created a new channel, we can invoke it to confirm that it works as expected.

% curl localhost:17010/api/v1/user/my.username ; echo
{"email": "my.user@example.com", "user_type": "ABC"}
%

In server logs:

INFO - api.user.get-details - Returning details of `my.username`

Everything is fine and we can proceed.

Just one note about channels, remember that there can be many channels pointing to the same service - this lets you reuse the services in various scenarios, e.g. you can have a separate REST channel with its own security definition for each external application connecting to your APIs.

Whenever you add, modify or delete a channel, the service as such is unchanged, nor are any other channels. Conversely, when you update a service, for instance adding new functionality, all channels using it will automatically invoke the new version of the service.

Channels come with good defaults when you create them but it is always possible to customize them to one's particular needs. Let's go step by step through each attribute of a channel's definition.

Basic information

Name - Each channel has a unique name which can be arbitrary, depending on your naming conventions.
Active - An inactive channel cannot be invoked; doing so will yield a 404 error.

URL path & query string

URL path - URL paths need to be unique. Each can contain one or more patterns to match input parameters - this is why in our service we were able to reference self.request.input.user_name - it was extracted from the URL path.
Match slash - Sometimes, URL path parameters sent from API clients will contain the slash character which normally is used to separate path components. This checkbox controls whether path patterns should match a slash or not.
URL params - QS over path vs. Path over QS. Usually, applications will send parameters in one place only, e.g. only in the URL path or in query string parameters. But what if an application has good reasons to send parameters in both the query string and URL path, for instance /api/v1/user/my.username?user_name=my.other.username - this happens from time to time and this setting lets one control which of the two will take precedence.
Merge to request - Used if parameters are sent via query string in addition to the URL path. If checked, such query string parameters will be accessible through self.request.input, otherwise, they will exist only in self.request.http.GET.
Params priority - This is similar to the options above but it will control the behavior if parameters are sent in both JSON message body as well as in URL path or query string - it lets one decide which will have higher priority and which will become available in self.request.input.

HTTP metadata & data format

Method - it is possible to specify that only specific HTTP method can be used to invoke the channel. This is not always needed. For instance, the service above can work just fine with either GET or POST, depending on what the caller prefers. Conceptually, this is a GET request but some API clients will always use POST so this is why filling out this field is not always needed or advised.

A service can handle multiple methods either as a whole and a channel can dispatch requests to specific parts of the service depending on which method is used, i.e. you can add handle_GET, handle_POST and other such methods independently, resulting, for instance, in a User service that handles CRUD via respective handle_* methods.

Encoding - Can be set to "gzip" to enable compression of responses from the channel.
Data format - Can be one of JSON, XML or HL7 to enable auto-de/serialization of requests and responses. Note that the original request, prior to any processing, is always available in self.request.raw_request.
Accept header - To what Accept HTTP headers the channel should react. Note that, just like with methods, it is possible for the service to handle each header separately, which lets one have the same service produce different responses depending on what the client's Accept header dictates.

API services & security

Service - The service that is mounted on this channel and which will be given incoming requests on input, as in the code example above. The same service may be mounted on multiple channels.
Security definition - What kind of an authentication mechanism this channel requires. Can be one of Basic Auth, API key, JWT, Vault, WS-Security or XPath. Each channel may have a different security definition assigned and the same definition can be assigned to multiple channels. Note that, to avoid any misunderstandings, you need to explicitly choose "No security definition" if the channel should not handle authentication itself, perhaps because the service should do it on its own, i.e. it is not possible to forget to choose something here and leave a channel unsecured by mistake.
RBAC - A channel can also use Role-Based Access Control definitions where each API client is assigned fine-grained permissions to individual methods and roles, possibly hierarchical one, which control if a particular client can invoke the channel's service. For instance, some API clients may be allowed to only read (GET) data whereas different ones will be able to create, update and delete it too (POST, PATCH and DELETE).

API caching

Cache type - Optionally, can be set to either built-in or Memcached. Requests matching previous ones will be automatically served from a specific cache assigned to the channel. With the built-in cache, the response is served from RAM directly, resulting in very fast responses. It is possible to browse, modify or delete the cached data, as below:

API rate limiting

The rate limiting stanza was not expanded previously so let's do it now to add a rate limiting definition to our channel. As usual, the limits can be set separately for each channel.

In this particular case, clients connecting from the internal network will be allowed to issue 30 request per minute but connections from localhost can invoke it 50k times an hour. And everyone else is limited to 100 requests a day.

Message log

Just like with rate limiting, these options were not expanded in the initial screenshot so here they are. Each channel can keep N last messages received, sent or both. It is also possible to say that only the first X kilobytes of a given message are to be stored - some messages may be too large for it to be practical to keep them in their entirety. When the log becomes full, the oldest messages are discarded, making room for newer ones.

Stored messages can be browsed in web-admin, immediately letting one know what a given channel receives and sends.

Such a message log is a feature common to other elements of Zato, e.g. HL7 or WebSocket connections have their own message logs too.

Documentation and OpenAPI

It is good that we have a REST channel but how do we let others know about it? On the one hand, you can simply notify your partners and clients through documentation, wiki pages or other non-automated means.

And by the way, if you need a WSDL for SOAP clients, the generated documentation includes it too.

Invoking REST APIs

This is everything about REST channels today but one question may be still open. If channels are means for handling incoming connections then how does one make requests to REST API endpoints external to Zato? How to invoke other other people's microservices?

The answer is that this is what Zato outgoing connections are for and we will cover REST outgoing connections in a future post in a way similar to how we went through REST channels in this one.

Understanding the fan-out / fan-in API integration pattern

Mon, 21 Dec 2020 03:00:00 +0000

Understanding the fan-out / fan-in API integration pattern

2020-12-21, by Dariusz Suchojad

What is fan-out / fan-in?

The core idea is this:

A source system requests some kind of information from your Zato service
The source system understands that the process of gathering this information may take time. This may be because the backend systems are slow to reply or there are many of them and the person using the source system will not actively wait for the reply, e.g. no one will want to wait 30 or 60 seconds in such a case.
Zato accepts the initial request and returns to the initial system a correlation ID (CID), a random string that tells the system "I received your request, here, save this CID somewhere, and when I have the result, I will give it to you along with that CID to make it possible for you to understand a reply to which request it is"
At this point the initial system knows that the response will arrive at some point in the future and it stores the CID in a datastore of choice, it can be in Redis, MongoDB, an SQL database or simply in RAM, this is completely up to the system
The system does not block the connection it made to Zato anymore now - this is important - it does not wait for the response to be sent as part of the same TCP exchange of messages
What exactly the system does instead of blocking depends on the protocol used to connect with Zato. If it is HTTP, it can possibly close the connection in the knowledge that Zato will invoke one of its callback endpoints once the results are ready. If it is AMQP, IBM or Zero MQ, then the system already listens for data in some reply queues.

The above concludes the initial phase and Zato can now carry out the rest of the work:

Your Zato service uses self.patterns.fanout to invoke as many data sources as required to collect the information originally request
A data source may mean a database of any sort, an external application or perhaps another service that in turn may invoke other applications, systems, services or fan-out / fan-in patterns as well
The service initially using self.patterns.fanout tells the pattern what callback service to invoke once all the data sources returned their responses. The pattern is also given the CID so that the callback service knows what it was too.

Now, the data-collecting services and data sources do their job. What it means is purely up to a given integration process, perhaps some SQL databases are consulted, maybe Cassandra queries run and maybe a list of REST endpoints is invoked, any protocol or data format can be used.

What Zato does in the meantime is keeping track of which of the data sources already replied and when they all have, Zato will invoke the callback service
The callback service is given on input all the responses along with the original CID
What the callback service will process the responses, e.g. by transforming them into a common format combining them into a single message
Finally, the callback service invokes the callback endpoint that the initial system awaits on - for instance, it may publish the response to an AMQP queue or it may invoke a REST endpoint. Note that the technology used here can be independent of the one used initially - as an example, we can very well have the very first system use WebSockets for requests and IBM MQ for responses.

Thus, we can see why it is called fan-out / fan-in:

First, the flow of information fans out to applications and systems containing the data needed
Next, the flow fans in through Zato to the application that requested the data

When to use it?

The pattern can be considered each time there is a rather long-running integration process that requires communication with more than one system.

This raises a question of what is long-running? The answer lies in whether there is a human being waiting for the results or not. Generally, people can wait 1-2 seconds for a response before growing impatient. Hence, if the process takes 30 seconds or more, they should not be made to wait as this can only lead to dissatisfaction.

Yet, even if there is no real person for the result, it is usually not a good idea to wait for results too long as that is a blocking process and blocking may mean consuming up resources unnecessarily, which in turn may lead to wasted CPU and, ultimately, energy.

This kind of processes is found when the data sources are uneven in the sense that some or all of them may use different technologies or because they may each return responses with different delays. For instance:

Some data sources may include reporting databases that generate reports on demand - such an act may require many seconds or minutes to complete
Some data sources may belong to different organizational units, e.g. an association of independent business organizations may request data from some of them and each may use its own system, database or endpoint, each with different a response time

Another trait of such processes is that the parallel paths are independent - if they are separate and each potentially takes a significant amount of time then we can just as well run them in parallel rather than serially, saving in this way time from the caller's perspective.

How does it look like in Python?

An example service using the pattern may look like below.

The service receives a user_id on input and is to return a credit scoring based on that
Three services, each representing a single backend system, are invoked in parallel
Once all the responses are available, a callback service is invoked - this is the service that needs to deliver the combined response to the original caller

Note that this particular code happens to use AMQP in the callback service but this is just for illustration purposes and any other technology can be used just as well.

from zato.server.service import Service

class GetUserCreditScoring(Service):

    def handle(self):

        user_id = self.request.input.user_id

        # A dictionary of services to invoke along with requests they receive
        targets = {
            'crm.get-user': {'user_id':user_id},
            'erp.get-user-information': {'USER_ID':user_id},
            'scoring.get-score': {'uid':user_id},
        }

        # Invoke all the services and get the CID
        cid = self.patterns.fanout.invoke(targets, callbacks)

        # Let the caller know what the CID is
        self.response.payload = {'cid': cid}

class GetUserCreditScoringAsyncCallbac(Service):

    def handle(self):

        # AMQP connection to use
        out_name = 'AsyncResponses'

        # AMQP exchange and routing key to use
        exchange = '/exchange'
        route_key = 'zato.responses'

        # The actual data to send - we assume that we send it
        # just as we received it, without any transformations.
        data = self.request.raw_request

        self.outgoing.amqp.send(data, out_name, exchange, route_key)

More options

Be sure to check details of the pattern in the documentation to understand what other options and metadata are available for your services.

In particular, note that there is another type callback not shown in this article. The code above has only one, final callback when all of the data is already available. But what if you need to report progress each of the sub-tasks completed? This is when per-invocation callbacks come in handy, check the samples in the documentation for more information.

More patterns

Fan-out / fan-in is just one of the patterns that Zato ships with. The most prominent ones are listed below:

Parallel execution - Similar to fan-out / fan-in in that it can be used for communication with multiple systems in parallel, the key difference being that this pattern is used when a final callback is needed, e.g. when there is a need to update many independent systems but there is no need for a combined response
Invoke/retry - used when there is a need to invoke a resource and handle potential failures in a transparent way, e.g. a system may be down temporarily and with this pattern it is possible to repeat the invocation at a later time, according to a specific schedule

Configuring REST channels for CORS

Mon, 14 Dec 2020 03:00:00 +0000

Configuring REST channels for CORS

2020-12-14, by Dariusz Suchojad

Terminology

CORS, as an acronym, has several parts:

Cross-Origin
Resource
Sharing

In the context of REST APIs, which predominantly means JSON or XML served usually with POST or other calls, this can be simplified to the below.

Origin means a combination of a URL scheme, hostname and port number. For instance, https://example.com is one origin whereas https://example.com:8443 is a different one (ports differ). Similarly, https://www.example.net is different than https://api.example.net (subdomains differ). Also, http:// vs. https:// is a different URL scheme and that means a different origin too even if domains and ports were the same.

Resource, here, is a REST API endpoint that is accessible via HTTP to JavaScript calls such as XMLHttpRequest (XHR). Note that XHR is not the only mechanism used but, because it is the most popular one in practice today, this article will concern with XHR only.

Sharing is the act of making resources available to XHR calls.

Finally, we deal with cross-origin sharing which means that JavaScript code and the REST endpoint will be served from different origins. E.g. https://www.example.com for the frontend where XHR calls come from whereas https://api.example.com is used for the REST endpoints.

Putting it together, in the most commonly found case, we have a frontend web page served from one domain and a REST API served from another domain and we want for the frontend to make XHR calls to the REST endpoints. CORS is the technique employed to make it happen. Without CORS, we would encounter a restriction called Same-Origin Policy (SOP) which would prevent the calls from succeeding.

However, note that the restriction does not apply to, for instance, resources loaded via <script src="..."> elements. This is why libraries like JQuery can be hot-linked to using CDNs or other locations but regular calls to one's REST APIs may require CORS configuration. Likewise, browser extensions may be exempt from SOP.

In short, in CORS, we deal with XHR and similar mechanisms specifically and other features that browsers have to offer may require it or not - do not be surprised if SOP and CORS are used in one place but not in another, outside of the realm of REST APIs invoked from pages served by web servers which is what this article covers.

Also, note that it is the web browser that CORS and SOP are enforced by - it means that if an endpoint is accessed via another HTTP client, such as curl, the restrictions will not apply.

Two types of CORS calls in REST APIs

A browser will issue two types of XHR calls:

Simple ones, usually not used in REST APIs built around JSON or XML
Complex ones, called pre-flighted ones, most often used with JSON or XML

With simple calls, an XHR call is made and certain specific headers will need to exist for the XHR caller to be able to read the response from the API server.

With complex ones, the browser first sends a short OPTIONS request (pre-flight), asking the server if the actual request can be sent. If the response is satisfactory, the browser will send a second request, this time with the real data that the XHR user meant to send.

Note that CORS is handled by browsers automatically. For instance, when an XHR POST call is issued, it is the browser that sends OPTIONS under the hood and the frontend developer does not control whether it happens or not.

Thus, the server's job is to return headers expected in a given type of a CORS-using invocation. The headers tell the browser whether it can proceed and if so, the browser continues in the process, ultimately returning the response from an API call to JavaScript code.

Keep in mind that, from the browser's perspective, CORS is not about preventing an XHR caller from invoking specific APIs. Rather, it is about preventing responses to such calls from reaching to the XHR caller. This means that in server logs you may notice GET, OPTIONS, POST or other invocations even if the browser does not deliver them to JavaScript.

At this point, it is best to read the Mozilla article about CORS. It goes into all the details and, in fact, given that it is concerned with the web as a whole, some parts of it do not translate directly into REST APIs and Zato but the knowledge is required nevertheless to make sure that the configuration is correct.

The rest of this blog post will assume that you now know what to configure and only a way to do it with Zato needs to be shown.

Configuring Zato for CORS

Configuration of Zato services can be condensed to two main choices:

We can use the after_handle response hook
We can add handle_OPTIONS

Consider the code below. It uses after_handle - this is a method that is invoked each time the handle method has just finished. We can take advantage of it to inject response HTTP headers.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class MyService(Service):

    def handle(self):
        # Business implementation goes here
        pass

    def after_handle(self):

        # We only allow requests from this particular origin
        allow_from_name = 'Access-Control-Allow-Origin'
        allow_from_value = 'https://www.example.com'

        self.response.headers[allow_from_name] = allow_from

Now, the same but using handle_OPTIONS.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class MyService2(Service):

    def handle(self):
        # Business implementation goes here
        pass

    def handle_OPTIONS(self):

        # We only allow requests from this particular origin
        allow_from_name = 'Access-Control-Allow-Origin'
        allow_from_value = 'https://www.example.com'

        self.response.headers[allow_from_name] = allow_from

The difference between the two is not large. Mainly, after_handle runs regardless whether the channel is invoked using REST, SOAP, IBM MQ, AMQP, WebSockets or any other possible which means that in some cases it may be superfluous.

On the other hand, with REST channels, after_handle runs no matter what input HTTP verb the request is using while handle_OPTIONS reacts to OPTIONS only so after_handle may be used for both simple and pre-flight CORS configuration.

All things considered, which one to use will be a choice to make based on a given integration scenario - both are applicable, depending on specific requirements.

Note also that in the code above each service has its own CORS configuration. Yet, it is likely that the same configuration should be shared by multiple services. Simple Python inheritance will work here nicely:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class BaseService(Service):

    def after_handle(self):

        # We only allow requests from this particular origin
        allow_from_name = 'Access-Control-Allow-Origin'
        allow_from_value = 'https://www.example.com'

        self.response.headers[allow_from_name] = allow_from

class MyChildService1(BaseService):

    def handle(self):
        # Business implementation goes here
        pass

class MyChildService2(BaseService):

    def handle(self):
        # Business implementation goes here
        pass

Or, with handle_OPTIONS:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class BaseService(Service):

    def handle_OPTIONS(self):

        # We only allow requests from this particular origin
        allow_from_name = 'Access-Control-Allow-Origin'
        allow_from_value = 'https://www.example.com'

        self.response.headers[allow_from_name] = allow_from

class MyChildService1(BaseService):

    def handle(self):
        # Business implementation goes here
        pass

class MyChildService2(BaseService):

    def handle(self):
        # Business implementation goes here
        pass

Now, the same configuration is accessible to more than one service and if anything needs to be changed, it is done in the base service.

Observe that the code above uses 'Access-Control-Allow-Origin' only but CORS employs other headers as well - you just put them all in self.response.headers, it is a regular Python dictionary with keys mapping to HTTP header names and values to header values.

This is everything that is needed to configure Zato for CORS-using REST API calls - choose the method most suitable (after_handle or handle_OPTIONS), assign headers to self.response.headers, hot-deploy your service and that is it!

Multi-protocol file transfer integrations

Mon, 07 Dec 2020 03:00:00 +0000

Multi-protocol file transfer integrations

2020-12-07, by Dariusz Suchojad

In many domains, transfer of static and batch files is an important part of systems integrations and a large number of applications produce and expect data in the form of files rather than network-based APIs. In this article, we shall see how Zato makes multi-protocol integrations of this kind possible in a way that is secure, scalable and easy to extend in Python.

File transfer is often found in scenarios such as:

Mass-transfer of invoices, e.g. an ERP may produce an hourly list of new ones to process
Batch-transfer of patient data in healthcare systems
Publication of updates to data such as stocks or exchange rates
Secure exchange of orders with external business partners, e.g. using files encrypted with PGP

There are many more scenarios but what they all have in common is that in each such data flow there may be a single producer and multiple recipients of information, each of them preferring to use different technologies.

For instance, a single file saved to a Windows share may go to a REST endpoint, an IBM MQ or AMQP queue, an AWS S3 bucket and to e-mail recipients, each of which may be an independent application or business entity and each may use a different data format, e.g. what is CSV initially may need to be transformed to JSON, XML and PDF, depending on the receiver.

Let's see how Zato helps in implementation of processes of this kind.

File transfer channels

The core Zato concept in file integrations is that of file transfer channels - reusable definitions pointing to producers and consumers of data along with accompanying details, such as a schedule, file patterns to use, what encoding the data is in and more.

Zato will connect to the data source automatically and handle each new or modified file according to the channel's configuration, i.e. by parsing and distributing it to data recipients, allowing one to apply any transformations or enrichment as required, and optionally deleting the data file from the source location afterwards.

In the Zato Dashboard, a file transfer channel may be defined as below.

Let's consider the following sample definition. In this particular case:

The data source is an SFTP connection definition called CRM
New or changed files are processed according to a schedule configured for a job called Transfer Invoices Job
Multiple input directories and glob file patterns can be used
Among of data recipients are REST outgoing connections - any file picked up from the source location will be sent to a REST endpoint directly as-is, without any programming needed
A file can be sent to [a service]/en/docs/4.1/intro/esb-soa.html) - and the service may transform or enrich the file as needed, e.g. it may change its data format, invoke more APIs to obtain additional data, combine it all and send the result to one or more recipients

Note the flexibility and reusability - the same data source, schedule and recipients can be configured to take part in multiple file integration processes without influencing each other.

In fact, the same services and endpoints may independently participate in both file and online API integrations.

Sample process

To showcase how file transfer works in practice, let's implement an example process:

An external application periodically produces customer orders as CSV
The CSV files are made available on SFTP that Zato can access
They are to be checked once a minute
Each file needs to be sent to a REST endpoint as-is, an AMQP queue as JSON and as an Excel e-mail attachment to selected people

Initial Dashboard configuration

We need to configure a few Zato objects for the new file transfer to use, check below for details of how to do it:

An SFTP connection
A scheduler's job
A REST connection definition
An AMQP connection definition
An SMTP connection definition

Simply fill out a form such as the ones below, click OK and you are done. No server restarts are needed.

REST recipients are special

Because so many new integrations are based around REST, recipients of this kind are treated specially by file transfer channels in that there is no programming required for files to be transferred to REST endpoints.

When you create a file transfer channel, simply assign one or more REST endpoints to it and Zato will send all the files to them as soon as they are picked up from the data source.

As to other recipient types, we will use the Python-based service below.

Python code

Let's quickly implement a service that will transform and send the files to AMQP and SMTP recipients.

The service will receive its input in an attribute called self.request.raw_request - we will transform it to the required data formats and send to both AMQP and SMTP.

# -*- coding: utf-8 -*-

# stdlib
import io

# For Excel files
from openpyxl import Workbook
from openpyxl.writer.excel import save_workbook

# Zato
from zato.common.api import SMTPMessage
from zato.common.json_ import dumps
from zato.server.service import Service

# ##############################################################################

class HandleNewOrders(Service):
    name = 'api.file-transfer.handle-new-orders'

    def handle(self):

        # All of our input data, including metadata assigned by Zato
        request = self.request.raw_request

        # Full path, including directory and file name
        full_path = request['full_path']

        # Channel the file was received through
        channel_name = request['channel_name']

        # Data received, already parsed by our channel
        data = request['data']

        # Log info that we are ready
        self.logger.info('Processing file `%s` from `%s`', full_path, channel_name)

        # Send to the CRM using AMQP first ..
        #self.send_orders_to_crm(data)

        # Send an email now ..
        self.notify_new_orders(data)

        # We are done!

# ##############################################################################

    def send_orders_to_crm(self, data):
        """ Converts input orders to JSON and sends them to the CRM using AMQP.
        """
        # A list of orders to be sent as JSON
        orders = []

        # Process each input CSV row ..
        for row in data:

            # .. extract information from each line ..
            order_no = row[0]
            customer_id = row[1]
            order_amount = row[2]

            # .. create a dict representing each order ..
            order = {
                'order_no': order_no,
                'customer_id': customer_id,
                'order_amount': order_amount,
            }

            # .. append the dict to already existing ones.
            orders.append(order)

        # Convert the list of orders into a JSON string
        json_data = dumps(orders)

        # Details of the AMQP connection to use ..
        conn_name = 'My AMQP Connection'
        exchange = '/orders.new'
        routing_key = 'zato'

        # .. and now, we can use the already converted message using AMQP.
        self.outgoing.amqp.send(json_data, conn_name, exchange, routing_key)

# ##############################################################################

    def notify_new_orders(self, data):

        # Create a new workbook ..
        workbook = Workbook()

        # .. create a new worksheet for orders ..
        sheet = workbook.create_sheet('New orders')

        # .. iterate over all CSV orders ..
        for row_idx, row in enumerate(data, 1):

            # .. now, iterate over all columns ..
            for column_idx, value in enumerate(row, 1):

                # .. create a new cell in the sheet ..
                sheet.cell(column=column_idx, row=row_idx, value=value)

        # .. our in-memory buffer for the Excel file ..
        buffer = io.BytesIO()

        # .. save the workbook to RAM ..
        save_workbook(workbook, buffer)

        # .. rewind to the beginning ..
        buffer.seek(0)

        # .. get the Excel file ..
        excel_data = buffer.read()

        # .. do not forget to close the buffer now that we do not need it.
        buffer.close()

        # We have the Excel file so we can send it as an e-mail attachment now.

        # In practice, this would be kept in a config file,
        # Redis, SQL or another config source.
        email_to = 'hello@example.com'
        email_cc = ['my.cc.1@example.com', 'my.cc.2@example.com']
        email_from = 'zato@example.com'
        email_body = '<h1>New orders</h1><br/>New orders are in the attachment.'
        file_name = 'new_orders.xlsx'

        # Construct a new message ..
        message = SMTPMessage()

        # .. we send an HTML file ..
        message.is_html = True

        # .. assign details ..
        message.to = email_to
        message.cc = email_cc
        message.from_ = email_from
        message.body = email_body

        # .. attach the file ..
        message.attach(file_name, excel_data)

        # .. obtain a connection to the SMTP server ..
        conn = self.email.smtp.get('My SMTP Connection').conn

        # .. and finally, send the message across.
        conn.send(message)

# ############################################################################

Creating a new file transfer channel

Now, we can create a new file transfer channel. After clicking OK, it will start to work.

If there is anything that needs changing, click Edit in web-admin and enter new information, e.g. new directories to look for files in.

If anything changes in any of the pieces of configuration that the channel uses, such as AMQP, REST or anything else, the channel will reconfigure automatically, without a need for a manual update.

Summary

Here we are, using Zato, we have just created a multi-protocol file transfer hub, integrating SFTP, AMQP and e-mail.

New protocols and recipients can be added at any time without disrupting existing file integrations - we can easily extend the service to transform and push the input files to SQL, Cassandra and Redis, for instance.

And if new data sources are needed, just create new channels, start them in addition to the existing ones and they will begin to work immediately. It is just that easy!

Backing up and restoring Zato Single Sign-On data

Fri, 29 May 2020 03:00:00 +0000

Backing up and restoring Zato Single Sign-On data

2020-05-29, by Anielkis Herrera

This article presents a procedure for backing up all of Zato Single Sign-On (SSO) data and restoring it later on.

A single Zato server with SQLite is used for simplicity reasons but the same principles hold regardless of the size of one's environment or the SQL database used.

Overview

There are two data sources that SSO uses:

Run-time information, such as users, groups, and all the other objects , are stored in an SQL database in tables prefixed with zato_sso_, e.g. zato_sso_user or zato_sso_group.
Encryption keys are kept in a file called secrets.conf - the same file is shared by all servers in a cluster

Thus, to make a backup:

Connect to an existing server via SSH
Dump the SQL contents of SSO tables and related objects such as indexes
Make a copy of secrets.conf
Save everything in a safe place

Conversely, to restore a backup:

Load the backup from the safe place
Connect to a new Zato server via SSH
Move the contents of the SQL dump to the database
Replace the server's secrets.conf with the one copied over earlier during backup

Backing up SQL data

Assuming that a Zato server is in a directory called /home/zato/sso1/server, here is how to back up an SQLite database:

$ cd /home/zato/sso1/server
$ sqlite3 zato.db ".dump zato_sso_%" > zato-sso-backup.sql

This will create a file called zato-sso-backup.sql the contents of which is the schema and rows of all the SSO objects.

To make it easier to restore the backup later, open the file and add the following commands right after "BEGIN TRANSACTION;"

DROP TABLE IF EXISTS zato_sso_group;
DROP TABLE IF EXISTS zato_sso_user;
DROP TABLE IF EXISTS zato_sso_user_group;
DROP TABLE IF EXISTS zato_sso_session;
DROP TABLE IF EXISTS zato_sso_attr;
DROP TABLE IF EXISTS zato_sso_linked_auth;
DROP TABLE IF EXISTS zato_sso_password_reset;

The idea with the DROP statements is that when you are restoring SSO from a backup, these tables, albeit empty, will already exist, so we can just drop them to silence out any SQLite warnings.

Backing up secrets.conf

Again, if the server is in /home/zato/sso1/server, the full path to secrets.conf is /home/zato/sso1/server/config/repo/secrets.conf - open the file and find a section called [secret_keys].

The section will have either one key called key1 or two keys, called key1 and sso_key1, depending on when your environment was created - older ones will only have key1 but newer ones have sso_key1 too.

If sso_key1 exists, Zato will use it to encrypt SSO data in the database, preferring it over key1. But if there is only key1 then this is what Zato uses.

Depending on which is available, make a copy of it. In a later step, it will be used as sso_key1 in a new environment. I.e. its new name will be always sso_key1, no matter if in the source environment it was key1 or sso_key1.

[secret_keys]
key1=P8ViJZs8hM...

Or:

[secret_keys]
key1=P8ViJZs8hM...
sso_key1=rmoz7btA8...

Creating a new server

We work under assumption that a new server will be created in a directory named /home/zato/sso2/server.

Note that it should be a completely new instance in a new cluster. Do not start the server yet.

Restoring SQL data

Move the zato-sso-backup.sql file to /home/zato/sso2/server and run the commands below:

$ cd /home/zato/sso2/server
$ sqlite3 zato.db < zato-sso-backup.sql
$ echo $?
$ 0

Exit code 0 should be returned on output, indicating a successful operation.

Using sso_key1 in the new server

The secret key copied from the original server (sso_key1 if it existed, key1 otherwise) can be used in the new server in several ways:

It can be added to the new server's secrets.conf:

[secret_keys]
key1=Tr4nTMQlh...
sso_key1=<insert the key here>

It can be provided as a command line argument to "zato start":

$ zato start /home/zato/sso2/server --sso-secret-key <insert the key here>

It can be exported to an environment variable and then used from command line (a variation of the second option):

$ export MY_SSO_SECRET_KEY
$ zato start /home/zato/sso2/server --sso-secret-key $MY_SSO_SECRET_KEY

If you use environment variables, ensure their names do not start with ZATO, case insensitively, they are reserved for system use.

Confirming it all

Now, the server can be started and we can confirm that the SSO data can be accessed by logging it to the system as one of its users, as below - output was reformatted for clarity:

$ zato sso login /home/zato/sso2/server my.user
User logged in {
  'username': 'my.user',
  'user_id': 'zusr6htg...',
  'ust': 'gAAAAABe0M_Pf8cdBa6bimnjfVUt5CF...',
  'creation_time': '2020-05-29T09:03:11.459337',
  'expiration_time': '2020-05-29T10:03:11.459337',
  'has_w_about_to_exp': False
}
$

That concludes the process - the SSO data is now restored and the server can be fully used, just like the original one.

Zato services as containers for Python functions and methods

Tue, 10 Mar 2020 03:00:00 +0000

Zato services as containers for Python functions and methods

2020-03-10, by Dariusz Suchojad

Acting as containers for enterprise APIs, Zato services are able to invoke each other to form higher-level processes and message flows. What if a service needs to invoke a hot-deployable Python function or method, though? Read on to learn details of how to accomplish it.

Background

Invoking another service is a simple matter - consider the two ones below. For simplicity, they are defined in the same Python module but they could be very well in different ones.

from zato.server.service import service

class MyAPI(Service):
    def handle(self):
        self.invoke('create.account', data={'username': 'my.user'})

class CreateAccount(Service):
    def handle(self):
        self.logger.info('I have been invoked with %s', self.request.raw_request)

MyAPI invokes CreateAccount by its name - the nice thing about it is that it is possible to apply additional conditions to such invocations, e.g. CreateAccount can be rate-limited.

On the other hand, invoking another service means simply executing a Python function call, an instance of the other service is created (a Python class instance) and then its handle method is invoked with all the request data and metadata available through various self attributes, e.g. self.request.http, self.request.input and similar.

This also means that at times it is convenient not to have to write a whole Python class, however simple, only to invoke it with some parameters. This is what the next section us be about.

Invoking Python methods

Let us change the Python code slightly.

from zato.server.service import service

class MyAPI(Service):
    def handle(self):
        instance = self.new_instance('customer.account')

        response1 = instance.create_account()
        response2 = instance.delete_account()
        response3 = instance.update_account()

class CustomerAccount(Service):

    def create_account(self):
        pass

    def delete_account(self):
        pass

    def update_account(self):
        pass

There are still two services but the second one can be effectively treated as a container for regular Python methods and functions. It no longer has its handle method defined, though there is nothing preventing it from doing so if required, and all it really has is three Python functions (methods).

Note, however, a fundamental difference with regards to how many services are needed to implement a fuller API.

Previously, a service was needed for each action to do with customer accounts, e.g. CreateAccount, DeleteAccount, UpdateAccount etc. Now, there is only one service, called CustomerAccount, and other services invoke its individual methods.

Such a multi-method service can be hot-deployed like any other which makes it a great way to group utility-like functionality in one place - such functions tend to be reused in many places, usually all over the code, so it is good to be able to update them at ease.

Code completion

There is still one aspect to remember about - when a new instance is created through new_instance, we would like to be able to have code auto-completion available.

If the other service is in the same Python module, it suffices to use this:

instance = self.new_instance('customer.account') # type: CustomerAccount

However, if the service whose methods are to be invoked is in a different Python module, we need to import for its name to be know to one's IDE. Yet, we do not really want to import it, we just need its name.

Hence, we guard the import with an if statement that never runs:

if 0:
    from my.api import CustomerAccount

class MyAPI(Service):
    def handle(self):
        instance = self.new_instance('customer.account') # type: CustomerAccount

Now, everything is ready - you can hot-deploy services with arbitrary functions and invoke them like any other Python function or method, including having access to code-completion in your IDE.

Generating API documentation

Wed, 12 Feb 2020 03:00:00 +0000

Generating API documentation

2020-02-12, by Dariusz Suchojad

A recurring need in larger integration projects is generation of API documentation for users belonging to different, yet related, target groups. Read on to learn how to generate Zato-based API specifications for more than one group from a single source of information.

A typical scenario is granting access to the same APIs to external and internal users - what they have in common is that all of them may want to access the same APIs yet not all of them should have access to documentation on the same level of details.

For instance - external developers should only know what a given endpoint is for and how to use it but internal ones may also be given information about its inner workings, the kind of details that external users should never learn about.

If documentation for two such groups is kept separately, it may require more maintenance effort than necessary - after all, if most of it is the same for everyone then it would make sense to keep it in one place and only add or remove details, depending on which group particular API documentation needs to be generated for.

The previous article went through the process of generating API specifications step by step - this one goes even further by adding introducing the notion of tags that drive which parts of documentation to generate or not.

Python docstrings

Let us use one of the same services as previously - here is its basic form:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Int, Service

class RechargeCard(Service):
    """ Recharges a pre-paid card.
    Amount must not be less than 1 and it cannot be greater than 10000.
    """

    class SimpleIO:
        input_required = 'number', Int('amount')
        output_required = Int('status')

The result, when generated as Sphinx, will be like below:

Now, we would like to add some details that only internal users should have access to - this is how the docstring should be modified, assuming for a moment that there are two CRM systems in the company the usage of which depends on a particular end user's account ..

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Int, Service

class RechargeCard(Service):
    """ Recharges a pre-paid card.
    Amount must not be less than 1 and it cannot be greater than 10000.

    #internal

    For user accounts starting with QM - MyCRM is queried.
    For any other account - MyCRM-2 is queried.
    """

    class SimpleIO:
        input_required = 'number', Int('amount')
        output_required = Int('status')

.. with the corresponding change in the output:

What happened above was that we applied a tag called #internal - anything that follows it is included in the output only if that very tag is requested when generating the documentation.

Note that the tag name is arbitrary, it could be any other name. Note also that there may be more than tag in a docstring, e.g. #confidential, #private or anything else.

Command-line usage

To actually include the #internal tag, the zato apispec command needs to be told about it explicitly, as below:

$ zato apispec /path/to/server \
  --dir /path/to/output/directory \
  --include api.* \
  --tags public,internal

If you do not give the command any tags, only the public part of the docstring, one without any tags, will be included in the resulting documentation.

To make your clear to other developers, you can also directly use the tag public in the command - this will make it easier to understand that you want to generate publicly available information and nothing else.

$ zato apispec /path/to/server \
  --dir /path/to/output/directory \
  --include api.* \
  --tags public

Wrapping up

This is it, you are done now - you can keep your API documentation in one place now and include only the relevant parts of it depending on context - this ensures that no matter who the recipient of your documentation is, it will be always generated from a single source, thus ensuring that all the output will stay in sync.

Windows commands and PowerShell scripts as API microservices

Tue, 23 Jul 2019 03:00:00 +0000

Windows commands and PowerShell scripts as API microservices

2019-07-23, by Dariusz Suchojad

This post goes through the steps of exposing Windows commands and PowerShell scripts as remote Zato API services that can be invoked by REST clients.

This lets one access a fleet of Windows systems from a single place and makes it possible for Zato services to participate in Windows management processes.

Note that Zato servers always run on Linux and no installation of any kind of software under Windows is necessary for Zato to connect to remote systems.

Prerequisites

Start by installing a library that implements the remote Windows connectivity:

$ cd /opt/zato/current
$ ./bin/pip install pywinrm

Next, stop and start again any servers running, e.g.:

$ zato stop /path/to/server
$ zato start /path/to/server

Python code

Deploy the following service to your Zato cluster - note its name, windows.remote.management.

# -*- coding: utf-8 -*-

from __future__ import absolute_import, division, print_function, unicode_literals

# stdlib
from traceback import format_exc

# pywinrm
import winrm

# Zato
from zato.server.service import Service

class MyService(Service):
    name = 'windows.remote.management'

    class SimpleIO:
        input_required = 'type', 'data'
        output_required = 'exec_code'
        output_optional = 'status_code', 'stdout', 'stderr', 'details'
        response_elem = None
        skip_empty_keys = True

    def handle(self):

        # Local aliases
        input_type = self.request.input.type
        input_data = self.request.input.data

        # Validate input - we support either regular commands
        # or PowerShell scripts on input.
        if input_type not in ('cmd', 'ps'):
            self.response.payload.exec_code = 'error'
            self.response.payload.details = 'Invalid type'

        # Input was valid, we can try to execute the command now
        else:

            try:

                # Remote server details
                host = '10.151.139.17'

                # Credentials
                username = 'myuser'
                password = 'NZaIhMezvK00Y'

                # Establish a connection to the remote host
                session = winrm.Session(host, (username, password))

                # Dynamically select a function to run,
                # either for commands or PowerShell
                func = session.run_cmd if input_type == 'cmd' else session.run_ps

                # Run the function with input data given
                result = func(input_data)

                # Status code is always available
                self.response.payload.status_code = result.status_code
                self.response.payload.stdout = result.std_out
                self.response.payload.stderr = result.std_err

            except Exception:
                self.response.payload.exec_code = 'error'
                self.response.payload.details = format_exc()

            # Everything went fine
            else:
                self.response.payload.exec_code = 'ok'

REST channel

Create a new REST channel in web-admin and mount service windows.remote.management on it. Make sure to set data format to JSON.

Usage

Let us invoke the service from command line, using curl. For clarity, the output of commands below is limited to a few lines.

First, we will run a regular command to get a directory listing of drive C:

$ curl http://api:password@localhost:11223/windows -d '{"type":"cmd", "data":"dir c:"}'

{"status_code": 0,
 "exec_code": "ok",
 "stdout": " Volume in drive C has no label.\r\n
 Volume Serial Number is 1F76-3AB6\r\n\r\n
 Directory of C:\\Users\\Administrator\r\n\r\n07/22/2019  11:53 PM    <DIR>"}

What if we provide an invalid drive name?

$ curl http://api:password@localhost:11223/windows -d '{"type":"cmd", "data":"dir z:"}'

{"status_code": 1,
 "exec_code": "ok",
 "stderr": "The system cannot find the path specified.\r\n"}

Now, invoke a PowerShell script, which in this case is a single-line one to check connection from the remote Windows system to example.com, but it could be much more complex, there are no limitations:

$ curl http://api:password@localhost:11223/windows -d \
  '{"type":"ps", "data":"Test-Connection example.com"}'

This time, both stdout and stderr are returned but because the overall status_code is 0, we know that the invocation was successful.

{"status_code": 0,
 "exec_code": "ok",
  "stdout": "WIN-A3I92B... example.com     93.184.216.34",
  "stderr": "#< CLIXML\r\n<Objs Version=\"1.1.0.1\" ...",

The service is just a starting point and there are a couple ways to extend it:

Details of remote servers, including credentials, should be kept separately
Permissions, including ACLs, can be added to allow or disallow access to particular commands to selected REST users only

Yet, even in this simple form, it already shows how easy it is to connect to Windows servers and turn remote commands into REST APIs microservices.

On top of it, REST is but one of many formats that Zato supports - one could just as well design workflows around AMQP, ZeroMQ, IBM MQ, FTP or other protocols in addition to REST with no changes to Python code required.

MongoDB connections

Tue, 26 Mar 2019 03:00:00 +0000

MongoDB connections

2019-03-26, by Dariusz Suchojad

One of the newest additions in Zato 3.1 are MongoDB connections - learn in this article how to create and use them in your Python API services.

Dashboard

As with most parts of Zato, the most straightforward way to create a new connection definition is to fill out a form in web-admin.

Note that MongoDB connections pack in a lot of options and, to keep the user interface as lightweight as possible, only the most commonly used ones are displayed in the default view and the rest needs to be toggled as needed.

Pinging

To confirm that the connection is configured correctly, including credentials to connect with, you can ping it.

If everything is set up correctly, a response such as the one below will be returned.

Python API

A sample Python service may look like below - the client returned by self.out.mongodb is an instance of pymongo.MongoClient so anything that the PyMongo library can do is achievable from Zato too.

# -*- coding: utf-8 -*-

from __future__ import absolute_import, division, print_function, unicode_literals

# Zato
from zato.server.service import Service

class MyService(Service):
    def handle(self):

        # Obtain a MongoDB client
        client = self.out.mongodb['My MongoDB Connection'].conn.client

        # Select a database
        db = client.test

        # Insert a document
        db.my_collection.insert_one({'My object': 'My data'})

        # Get all documents already stored
        result = db.my_collection.find()

        # Print them out to logs
        for item in result:
            self.logger.info('Item: %s', item)

Now, in log files:

INFO - Item: {'_id': ObjectId('5c99f8aa5ecb8221dcb3ff73'), 'My object': 'My data'}
INFO - Item: {'_id': ObjectId('5c99ffdf5ecb8221dcb3ff74'), 'My object': 'My data'}
INFO - Item: {'_id': ObjectId('5c99ffe05ecb8221dcb3ff75'), 'My object': 'My data'}

Summary

This is everything that is required - once a connection is created it can be used immediately in Zato services.

Because each connection is actually an instance of the official MongoDB Python client, there are no limits to what can be done with them - building MongoDB-based applications with Zato is now a fully supported possibility.

HTTP Method and Accept headers

Mon, 18 Mar 2019 03:00:00 +0000

HTTP Method and Accept headers

2019-03-18, by Dariusz Suchojad

Zato 3.1 includes new means to manage access to REST services based on input Method and Accept headers in HTTP requests - here is how they can be employed in practice.

A bit of background

Prior to Zato 3.1, one could always build a REST API reacting to individual HTTP verbs by implementing handle_VERB methods in services, such as:

class MyService(Service):

    def handle_GET(self):
        # Reacts to GET requests
        pass

    def handle_POST(self):
        # Reacts to POST requests
        pass

    # Any other handle_VERB method will be used accordingly

This works and will continue to work as expected in all future Zato versions.

However, one aspect of it is that, if one uses SimpleIO, keeping all handler methods in the same service means that all of them share the same SIO definition which is not always desirable - for instance, input to POST may be unrelated to input that DELETE receives.

REST channel URL paths

In Zato 3.1 and newer, it is possible to create REST channels that have the same URL path but different services mounted on each channel, separately for each HTTP verb requires.

That is, whereas previously it was a single service with multiple handle_VERB methods, now it can be a set of services, each reacting to a different HTTP verb and all of them mounted on the same URL path.

In a way, this was supported previously but, if handle_VERB methods were not used, URL paths had to be distinct, e.g.

GET /api/user
DELETE /api/user/delete
POST /api/user/create

In 3.1+, this can be simplified to:

GET /api/user
DELETE /api/user
POST /api/user

Now, each of the combination of verb + path may be unique for a REST channel while previously each channel needed to have its own URL path.

Moreover, because each channel may have its own separate service, it also means that each service may have its own SimpleIO definition and that the service becomes less tied to REST.

HTTP Accept headers

This is a completely new feature in 3.1 which lets one have distinct REST channels depending on the requests's Accept headers.

For instance, let's say that we would like to process incoming invoices under POST /api/invoice but we would like to have two services reacting to the same endpoint, one for JSON and now for PDF invoices.

This can be achieved by configuring HTTP Accept headers in their channels, as below - note that the method and URL path are the same in both cases yet HTTP Accept and services are different because each service reacts to a different value of HTTP Accept:

HTTP Accept header patterns

We can go one better and take advantage of Accept header patterns - with an asterisk meaning any character - this will configure the channel to process requests matching any value that fits in with the pattern, e.g. text/ will mean text/csv, text/xml or anything that starts with text/*.

However, seeing as it can be a number of input MIME types, at this point we may need to know what the actual value was - this can be extracted from the WSGI environment via self.wsgi_environ.

Summary

Zato 3.1 saw improvements and backward-compatible changes to how REST channels can be built.

It will now support more use-cases, such as single URL path channels with different HTTP verbs and independent SimpleIO definitions or HTTP Accept headers different for each channel.

In turn, this lets one build REST APIs that are more flexible and elastic in their design to react to different input criteria.

Two-factor authentication in Zato Dashboard

Mon, 25 Feb 2019 03:00:00 +0000

Two-factor authentication in Zato Dashboard

2019-02-25, by Dariusz Suchojad

This post provides steps needed to enable and make use of TOTP two-factor authentication in the Zato Dashboard - the method is compatible with the most popular apps such as Google Authenticator, Authy or FreeOTP.

The functionality is slated for release in Zato 3.1 and, currently, it is available via Zato source installation.

Modifying web-admin.conf

We start by letting web-admin know that it is to require TOTP security codes from users. This is a global flag for all users.

Stop web-admin
Open web-admin.conf file
Set "is_totp_enabled" to true (note that it is lower-case, without quotes)
Save the file
Start web-admin back

Generating the initial TOTP key

Now that the two-factor authentication with TOTP is enabled, we need the initial TOTP key for each user, otherwise they will not be able to log in at all.

Command line can be used to reset such keys for any user - in this case the process of resetting a key will amount to setting the initial one, as in the sample below that resets the key for a web-admin user called admin:

$ zato reset-totp-key /path/to/web-admin admin
IDNSLY27ESRPWUI5
$

Such a key can now be transferred to an app of choice to generate time-based security codes that web-admin will require.

Logging in

The app that the key was saved to will now present security codes needed for logging in.

Each such key is valid for at most 30 seconds - in this way a potential attacker will have to take two factors into account, one is your password and the other is getting access to your app and the associated TOTP secret key. Replay attacks are not particularly feasible against such codes because they will change frequently.

As always, even without TOTP, you should also make sure you rate-limit login attempts to web-admin by configuring your frontend load-balancer, or proxy, accordingly. With TOTP, since the codes are relatively short, it would be possible for an attacker to check them all very quickly so this kind of rate-limiting is of double importance.

Changing your TOTP key in web-admin

Having logged in, it is possible to generate secret keys directly in web-admin too. This option will also output QR codes to transfer the code to mobile devices simply by scanning the resulting QR code.

Summary

That is all - everything is set up and two-factor authentication will be enforced each time anyone logs in. TOTP is easy to configure and use so it is recommended to enable it as an additional security layer.

Zato: A successful Python 3 migration story

Mon, 11 Feb 2019 03:00:00 +0000

Zato: A successful Python 3 migration story

2019-02-11, by Dariusz Suchojad

Now that Python 3 support is available as a preview for developers, this post summarizes the effort that went into making sure that Zato works smoothly using both Python 2.7 and 3.x.

In fact, the works required were remarkably straightforward and trouble-free and the article discusses the thought process behind it, some of the techniques applied or tools used.

Background

Zato is [an enterprise API integration platform and backend application server]/en/docs/4.1/intro/esb-soa.html). We support a couple dozen of protocols, data formats, several sorts of IPC and other means to exchange messages across applications.

In other words, on the lowest level, passing bytes around, transforming, extracting, changing, collecting, manipulating, converting, encoding, decoding and comparing them, including support for all kinds of natural languages from around the world, is what Zato is about at its core when it is considered from the perspective of the programming language it is implemented in.

The codebase is around 130,000 lines of code, out of which Python and Cython are 60,000 lines. This is not everything, though, because we also have 170+ external dependencies that need to work with Python 2.7 and 3.x.

The works took two people a total of 80 hours. They were spread over a longer calendar time, except for the final sprint that required more attention for several days in a row.

Preparations

Since the very beginning, it was clear that Python 3 will have to be supported at one day so the number one thing that each and every Python module has always had is this preamble:

from __future__ import absolute_import, division, print_function, unicode_literals

This is what every Python file contains and it easily saved 90% of any potential work required to support Python 3 because, among other less demanding things, it enforced a separation, though still not as strict as in Python 3, between byte and Unicode objects. The separation is a good thing and the more one works with Python 3 the clearer it becomes.

In Python 2, it was sometimes possible to mix the two. Imagine that there is a Python-derived language where JSON dicts and Python dicts can be sometimes used interchangeably.

For instance, this is a JSON object: {"key1": "value1"} and it so happens that it is also a valid Python dict so in this hypothetical language, this would work:

json = '{"key1": "value1"}'
python = {'key2': 'value2'}

result = json + python

Now the result is this:

{'key1': 'value1', 'key2': 'value2'}.

Or wait, perhaps it should be this?

'{"key1": "value1", "key2": "value2"}'

This is the central thing - they are distinct types and they should not be mixed merely because they may be related or seem similar.

Conceptually, just like upon receiving a JSON request from the network a Python application will decode it into a canonical representation, such as a dict, list or another Python object, the same should happen to other bytes, including ones that happen to represent text or similar information. In this case, the canonical format is called Unicode, and that is the whole point of employing it in one's application.

All of this was clear from the outset and the from __future__ statements helped in its execution, even if theoretically one could have been still able to mix bytes and Unicode - it was simply a matter of using the correct canonical format in a given context, i.e. a case of making sure the architecture was clean.

This particular __future__ statement was first announced in 2008 so there was plenty of time to prepare to it.

As part of the preparations, it is good to read a book about Unicode. Not just a 'Unicode for overburdened developers' kind of an article but an actual book that will let one truly appreciate the standard's breadth and scope. While reading it, do not resist the temptation to learn at least basics of two or more natural languages that you never knew about before. It will only help you develop into a better person and this is not a joke.

While programming with bytes and Unicode, it is convenient simply to forget about whether it is a 'str', 'bytes' or 'unicode' object - it is easier simply to think about bytes and text. There are bytes that can mean anything and there is text whose native, canonical form is Unicode. This is not always 100% accurate because Unicode can represent marvelous gems such as Byzantine musical notation and more but if a given application's scope is mostly constrained to text then this will work - there are bytes and there is text.

This is all fine with our own code but there are still the external libraries that Zato uses and some of them will want bytes, not text, or the other way around, in seemingly similar situations. There can be even cases like a library expecting for protocol header keys to be text and protocol header values to be bytes for rather unclear reasons. Simply accept it as a fact of life and move on with your works, there is no need to pause even for a moment to think about it.

Side projects

It was good to try out Python 3 first in a few new, smaller side-projects, GUI or command-line tools that are not part of the core yet they are important in the overall picture. The most important part of it was that creating a Python 3 application from scratch was in no way different than in Python 2, this served as a gentle introduction to Python 3-specific constructs and this knowledge was easily transferred later on to the main porting job.

Dependencies

Out of a total of 170+ dependencies, around 10 were not Python 3-compatible. All of them had not been updated in eight, twelve or more years. At this point, it is safe to assume that if there is a dependency that was last updated in 2009 and it has no Python 3 support then it never will.

What to do next depended on a particular case, each of them was some kind of a convenience library - sometimes they had to be dropped and sometimes forked. Most complex changes required in a fork were on the level of updating 'print' to 'print()' or doing away with complex installation setups that predated contemporary pip-based configuration options.

Other than that, there were no issues with dependencies, all of them were ready for Python 3.

Idioms and imports

Most of the reference information needed to make use of Python 2 and 3 was available via the python-future project which itself is a great assistance. Installing this library, along with its dependencies, sufficed for 99% of cases. There were some lesser requirements that were incorporated into a Zato-specific submodule directly, e.g. sys.maxint is at times useful as a loop terminator but ints in Python 3 have no limits so an equivalent had to be added to our own code.

Note that the page above does not show all the idioms and some changes were not always immediately obvious, like modifications to __slots__, or the way metaclasses can be declared, but there were no really impossible cases, just different things to use, either built in to Python 3 or available via future or six libraries.

A nice thing is that one is not required to immediately change all the imports in one go - they can be changed in smaller increments, e.g. 'basestring' is still available in the form of 'from past.builtins import basestring'.

Testing

A really important aspect during the migration was the ability to test sub-components of an application in isolation. This does not only include unittests, which may be too low-level, but also things such as starting only selected parts of Zato without a requirement to boot up whole servers which in turn meant each change could be tested within one second rather than ten. To a degree, this was an unexpected but really useful test of how modular our design was.

Intellectually, this was certainly the most challenging part because it required maintaining and traversing several trains of thought at once, sometimes for several days on end. This, in turn, means that it really is not a job for late afternoons only and it cannot be an afterthought, things can simply get complex very quickly.

String formatting

There is one thing that was not expected - the way str.format works with bytes and text.

For instance, this will fail in Python 3:

>>> 'aaa' + b'bbb'
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't convert 'bytes' object to str implicitly
>>>

Just for reference, in Python 2 it does not fail:

>>> 'aaa' + b'bbb'
'aaabbb'
>>>

Still under Python 2, let's use string formatting:

>>> template = '{}.{}'
>>> template.format('aaa', b'bbb')
'aaa.bbb'
>>>

In Python 3, this is the result:

>>> template = '{}.{}'
>>> template.format('aaa', b'bbb')
"aaa.b'bbb'"
>>>

In the context of a Python 3 migration, it would have been probably more in line with other changes to the language if this had been special-cased to reject such constructs altogether.

Otherwise, it initially led to rather inexplicable error messages because the code that produces such string constants may be completely unaware of where they are used further on. But witnessed once or twice, it was apparent later on what the root cause was and this could be easily dealt with.

Things that are missed

One small, yet convenient, feature of Python 2 was the availability of some of the common codecs directly in string objects, e.g.:

>>> u'abc'.encode('hex')
'616263'
>>> u'abc'.encode('base64')
'YWJj\n'
>>> u'ελληνική'.encode('idna')
'xn--jxangifdar'
>>>

This will not work as-is in Python 3:

>>> u'abc'.encode('hex')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
LookupError: 'hex' is not a text encoding; use codecs.encode() to handle arbitrary codecs
>>>

Naturally, the functionality as such is still available in Python 3, just not via the same means.

Python 2.7

On the server side, Python 2.7 will be around for many years. After all, this is a great language that let thousands and millions of people complete amazing projects and most of enterprise applications do not get rewritten solely because one of the technical components (here, Python) changes in a way that is partly incompatible with previous versions.

Both RHEL and Ubuntu ship with Python 2.7 and both of them have long-term support well into the 2020s so the language as such will not go away. Yet, piece by piece, all the applications will be changed, modified, modularized or rewritten and gradually Python 2.7's usage will diminish.

In Zato, Python 2.7 will be supported for as long as it is feasible and one of the current migration's explicit goals was to make sure that existing user Zato environments based on Python 2.7 will continue to work out-of-the-box with Python 3 so there is no difference which Python version one chooses - both are supported and can be used.

Summary

An extraordinary aspect of the migration is that it was so unextraordinary. There were no really hard-won battles, no true gotchas and no unlooked-for hurdles. This can be likely attributed to the facts that:

Core Python developers offered information what to expect during such a job
Unicode was not treated as an afterthought
Zato reuses common libraries that are all ported to Python 3 already
Internet offers guides, hints and other pieces of information about what to do
It was easy to test Zato components in isolation
Time was explicitly put aside for the most difficult parts without having to share it with other tasks