This GitHub repository contains Zato code developed during WSGI Wrestle 2013.

INTRO

It's a middleware cache for exchange rates data fetched using Yahoo! YQL's XML. It's meant to support hypothetical, yet practical, scenarios of storing backend data in a cache close to a frontend application. The client frontend is not included in the repository because Zato is an ESB and backend application server, not a frontend one.

The code can form part of a wider process of building analytical applications that need convenient access to aggregated data.

Splitting the functionality across multiple applications means higher reusability - Zato provides caching, API and data while frontend systems - no matter what technology they are written with - can focus on their own job of providing interesting and useful user interfaces.

A diagram

WHAT IT SUPPORTS AND DOES

  • Querying for an average exchange rate for a given date
  • Registering pairs of currency codes to periodically grab rates for (i.e. USD and EUR)
  • Deleting existing pairs
  • Listing all existing pairs
  • Periodically updating the cache with newest values read using YQL's XML response
  • Periodically trimming the cache so it doesn't grow indefinitely

All that is contained within a single Python module hot-deployed on servers running in a cluster behind a high-availability load-balancer.

SAMPLE USAGE

The installation documentation includes exposing two of the services over HTTP and JSON, so once everything is installed ...

Screenshot Screenshot

... curl can be used to register a couple of pairs and querying for their current exchange rate.

400: Invalid request

NOTES

  • Zato servers run in clusters that are built using gevent/gunicorn with HAProxy in front of them. This is completely transparent from a developer's viewpoint.

  • Services hot-deployed on one server in a cluster are automatically installed on other servers.

  • There are several services involved and each can be independently exposed over multiple access channels though the installation steps show only create-exchange-pair and get-rate.

  • update-cache and trim-cache modify the contents of the cache with a Redis-based distributed lock held. This is needed because each service may be possibly running in multiple copies on more than one server so there must be a way to ensure that each one has access to consistent data only - no partial updates must be visible.

  • dispatch-update-cache is a wrapper service which uses self.invoke_async to asynchronously invoke a separate instance of update-cache which actually updates the cache. Each instance may be running on different server and the load is spread automatically by Zato.

  • The solution developed can invoked from any frontend application using any programming language but for Python applications, Zato offers a Python client which allows one to communicate with Zato using regular Python dictionaries, as below

400: Invalid request

The client is built on top of the requests library and can be invoked from Django, Flask or any other Python code.

InfoQ have just published a Chinese translation of the original article (in English) that introduces all the major aspects of Zato - architecture, clusters, servers, load-balancer along with a sample financial integration with Yahoo/YQL and Google XML, GUI screenshots, CLI and an overview of all the user-visible features.

This is a straight-to-the-point read for team leaders, systems architects and developers.

Thanks to 马德奎 for the translation!

WebSphere MQ is a messaging middleware by IBM - a message queue server - and this post shows how to integrate with MQ from Python and Zato.

The article will go through a short process that will let you:

  • Send messages to queues in 1 line of Python code
  • Receive messages from queues without coding
  • Seamlessly integrate with Java JMS applications - frequently found in WebSphere MQ environments
  • Push MQ messages from Django or Flask

Prerequisites

Preliminary steps

  • Obtain connection details and credentials to the queue manager that you will be connecting to:

    • host, e.g. 10.151.13.11
    • port, e.g. 1414
    • channel name, e.g. DEV.SVRCONN.1
    • queue manager name (optional)
    • username (optional)
    • password (optional)
  • Install Zato

  • On the same system that Zato is on, install a WebSphere MQ Client - this is an umbrella term for a set of development headers and libraries that let applications connect to remote queue managers

  • Install PyMQI - an additional dependency implementing the low-level proprietary MQ protocol. Note that you need to use the pip command that Zato ships with:

# Assuming Zato is in /opt/zato/current
zato$ cd /opt/zato/current/bin
zato$ ./pip install pymqi
  • That is it - everything is installed and the rest is a matter of configuration

Understanding definitions, outgoing connections and channels

Everything in Zato revolves around re-usability and hot-reconfiguration - each individual piece of configuration can be changed on the fly, while servers are running, without restarts.

Note that the concepts below are presented in the context of WebSphere MQ but they apply to other connection types in Zato too.

  • Definitions - encapsulate common details that apply to other parts of configuration, e.g. a connection definition may contain remote host and port
  • Outgoing connections - objects through which data is sent to remote resources, such as MQ queues
  • Channels - objects through which data can be received, for instance, from MQ queues

It is usually most convenient to configure environments during development using web-admin GUI but afterwards this can be automated with enmasse, API or command-line interface.

Once configuration is defined, it can be used from Zato services which in turn represent APIs that Zato clients invoke. Then, external applications, such as a Django or Flask, will connect using HTTP to a Zato service which will on their behalf send messages to MQ queues.

Let's use web-admin to define all the Zato objects required for MQ integrations. (Hint: web-admin by default runs on http://localhost:8183)

Definition

  • Go to Connections -> Definitions -> WebSphere MQ
  • Fill out the form and click OK
  • Observe the 'Use JMS' checkbox - more about it later on

Screenshots

  • Note that a password is by default set to an unusable one (a random UUID4) so once a definition is created, click on Change password to set it to a required one

Screenshots

  • Click Ping to confirm that connections to the remote queue manager can be established

Screenshots

Outgoing connection

  • Go to Connections -> Outgoing -> WebSphere MQ
  • Fill out the form - the connection's name is just a descriptive label
  • Note that you do not specify a queue name here - this is because a single connection can be used with as many queues as needed

Screenshots

  • You can now send a test MQ message directly from web-admin after click Send a message

Screenshots

Screenshots

API services

  • Having carried out the steps above, you can now send messages to queue managers from web-admin, which is a great way to confirm MQ-level connectivity but the crucial point of using Zato is to offer API services to client applications so let's create two services now, one for sending messages to MQ and one that will receive them.
# -*- coding: utf-8 -*-

from __future__ import absolute_import, division, print_function, unicode_literals

# Zato
from zato.server.service import Service

class MQSender(Service):
    """ Sends all incoming messages as they are straight to a remote MQ queue.
    """
    def handle(self):

        # This single line suffices
        self.out.wmq.send(self.request.raw_request, 'customer.updates', 'CUSTOMER.1')
  • In practice, a service such as the one above could perform transformation on incoming messages or read its destination queue names from configuration files but it serves to illustrate the point that literally 1 line of code is needed to send MQ messages

  • Let's create a channel service now - one that will act as a callback invoked for each message consumed off a queue:

# -*- coding: utf-8 -*-

from __future__ import absolute_import, division, print_function, unicode_literals

# Zato
from zato.server.service import Service

class MQReceiver(Service):
    """ Invoked for each message taken from a remote MQ queue
    """
    def handle(self):
        self.logger.info(self.request.raw_request)

But wait - if this is the service that is a callback one then how does it know which queue to get messages from?

That is the key point of Zato architecture - services do not need to know it and unless you really need it, they won't ever access this information.

Such configuration details are configured externally (for instance, in web-admin) and a service is just a black box that receives some input, operates on it and produces output.

In fact, the very same service could be mounted not only on WebSphere MQ ones but also on REST or AMQP channels.

Without further ado, let's create a channel in that case, but since this is an article about MQ, only this connection type will be shown even if the same principle applies to other channel types.

Channel

  • Go to Connections -> Channels -> WebSphere MQ
  • Fill out the form and click OK
  • Data format may be JSON, XML or blank if no automatic de-serialization is required

Screenshots

After clicking OK a lightweight background task will start to listen for messages pertaining to a given queue and upon receiving any, the service configured for channel will be invoked.

You can start as many channels as there are queues to consume messages from, that is, each channel = one input queue and each channel may declare a different service.

JMS Java integration

In many MQ environments the majority of applications will be based on Java JMS and Zato implements the underlying wire-level MQ JMS protocol to let services integrate with such systems without any effort from a Python programmer's perspective.

When creating connection definitions, merely check Use JMS and everything will be taken care of under the hood - all the necessary wire headers will be added or removed when it needs to be done.

Screenshots

No restarts required

It's worth to emphasize again that at no point are server restarts required to reconfigure connection details.

No matter how many definitions, outgoing connections, channels there are, and no matter of what kind they are (MQ or not), changing any of them will only update that very one across the whole cluster of Zato servers without interrupting other API services running concurrently.

Configuration wrap-up

  • MQ connection definitions are re-used across outgoing connections and channels
  • Outgoing connections are used by services to send messages to queues
  • Data from queues is read through channels that invoke user-defined services
  • Everything is reconfigurable on the fly

Let's now check how to add a REST channel for the MQSender service thus letting Django and Flask push MQ messages.

Django and Flask integration

  • Any Zato-based API service can be mounted on a channel
  • For Django and Flask, it is most convenient to mount one's services on REST channels and invoke them using the zato-client from PyPI
  • zato-client is a set of convenience clients that lets any Python application, including ones based on Django or Flask, to invoke Zato services in just a few steps
  • There is a dedicated chapter in documentation about Django and Flask, including a sample integration scenario
  • It's recommended to go through the chapter step-by-step - since all Zato configuration objects share the same principles, the whole of its information applies to any sort of technology that Django or Flask may need to integrate with, including WebSphere MQ
  • After completing that chapter, to push messages to MQ, you will only need to:
    • Create a security definition for a new REST channel for Django or Flask
    • Create the REST channel itself
    • Assign a service to it (e.g. MQSender)
    • Use a Python client from zato-client to invoke that channel from Django or Flask
    • And that is it - no MQ programming is needed to send messages to MQ queues from any Python application :-)

Summary

  • Zato lets Python programmers integrate with WebSphere MQ with little to no effort
  • Built-in support for JMS lets one integrate with existing Java applications in a transparent manner
  • Built-in Python clients offer trivial access to Zato-based API services from other Python applications, including Django or Flask

Where to next? Start off with the tutorial, then consult the documentation, there is a lot of information for all types of API and integration projects, and have a look at support options in case you need absolutely any sort of assistance!

A feature that has been recently added to Zato in git and will be released in version 2.0 is an HTTP request profiler.

To start it, set profiler.enabled to True in server.conf:

400: Invalid request

Assuming a server is running on localhost using default settings, visiting http://localhost:17010/zato-profiler, like in the screenshot below, can quickly help you understand:

  • what functions and methods take the most time
  • what parts of code are most commonly used
  • who are the callers of a given function/method
  • what functions/methods given code is invoking

Screenshot

The profiler is exposed using the repoze.profile library which is a reusable WSGI middleware that can be embedded in other projects too.

Over at the #pyugat IRC channel - Python User Group Austria - there was a question how Zato handles XML namespaces given that working with SOAP doesn't require juggling any.

So for instance, given this SOAP request a typical Zato service can just freely access the elements without any particular effort.

400: Invalid request

400: Invalid request

400: Invalid request

The trick is, Zato uses lxml as its underlying XML parsing library and what can strike one as a smack of pure brilliancy, lxml.objectify's default mode of operation is to assume any child elements are in the same namespace their parent is.

It may seem nothing big at first but this is exactly the feature that makes working with SOAP in lxml as elegant (pythonic) as it can possibly get.

90% of time one will be working in the same namespace so why remember about it at all?

Yes, namespaces come in handy and yes, there can be multiple namespaces in one document but given that the prevailing majority of SOAP processing is to do with the single one namespace this particular document's business payload is in, why not forget about the whole thing?

Not having to deal with it at all was an excellent idea on lxml's part that greatly reduces time spent on development and improves the resulting code's readability, hence lowering the total maintenance costs.

The code above looks like accessing regular Python objects, or perhaps JSON. There is nothing really XML-specific to it.

This is what Zato actually does - it finds the first child in soapenv:Body, turns it into a Python object and makes it available to a service in self.request.input.

If you're coming to Zato or Python with background in other programming languages, this single feature will make SOAP processing a new experience for you.

What if you really need to access multiple namespaces? This, of course, is possible, as in the example below (which is a standalone program, not a Zato service, but the same thing can be done in Zato too).

400: Invalid request

400: Invalid request