While Zato can support AMQP, WebSphere MQ, ZeroMQ or FTP as well, the majority of environments will use the ubiquituous HTTP protocol only and this post describes the traffic patterns within a typical HTTP-only cluster consisting of a load-balancer, web admin, Redis, SQL ODB and two servers, right after it's been installed without any user-specific customizations.

Please refer to the Zato's architecture overview for a refresher on what role each of the components fulfills.

A diagram

  • In order to ensure high-availability (HA), incoming HTTP traffic is always routed through a load-balancer that distributes the load in a fair manner across the servers so that outside client applications don't access the servers directly and are never concerned if any of the servers is unavailable.

  • Servers always invoke external HTTP resources directly, from the systems they are installed on.

  • Servers never communicate directly with each other. If any server needs to send a message to another one, it's always done in an indirect manner using publish/subscribe with Redis.

  • Redis is also used by servers to store volatile or highly dynamic data, such as usage statistics, frequently accessed runtime configuration or user caches.

  • SQL ODB (Operational Database), which as of Zato 1.1 needs to be either PostgreSQL or Oracle, stores information that benefits from being kept in a relational database, usually data that needs to be always saved on disk as promptly as possible and shouldn't be stored in RAM first, as with Redis.

  • Web admin is a GUI used to manage Zato environments. Being based on Django, it connects to SQL ODB in order to store a few parts of its configuration but the vast majority of its functionality is merely a frontend that connects to servers, through the load-balanacer, to invoke the public API servers expose.

    In other words, this is a frontend only. Users are encouraged to use the API to create their own alternative frontend apps.

  • Arrowheads in the diagram represent the exact directions the traffic will follow - for instance, servers don't send messages to the load-balancer nor the latter will invoke Redis thus the arrows don't point in these directions.

Linux Journal has just published a free version of the Zato article originally written for the November 2013 issue.

The article presents the project, introduces the IRA philosophy of designing services that are:

  • Interesting
  • Reusable
  • Atomic and goes through the process of developing a sample integration using financial APIs of treasury.gov.

Please check it out here.

A diagram Cover Article

A Zato ESB and application server feature that has just been added on GitHub is an HTTP access log.

This is an additional log to what has been already available in regular server logs and keeps nothing but information regarding HTTP requests in a format used by Apache httpd. This makes it possible to use existing tools in order to parse and extract interesting information out of the logs.

The log is kept in each server's ./logs/http_access.log file. For instance, if a server is in /opt/zato/server1, the log will be in /opt/zato/server1/http_access.log.

Here's a sample entry and descriptions of all the default fields:

10.151.19.27 K070053855728592769928224227915635470962 "zato.ping" [12/Jan/2014:20:11:53 +0100] "GET /zato/ping HTTP/1.1" 200 141 "-" "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3"
Field Sample value Description
Remote address 10.151.19.27 Address of the application issuing the request, X-Forwarded-For is used, if present, otherwise IP address of the client application is used.
CID K070053855728592769928224227915635470962 Correlation ID assigned to the request
Channel name "zato.ping" Name of the channel invoked, either plain HTTP or SOAP
Timestamp [12/Jan/2014:20:11:53 +0100] Timestamp of the request. In local timezone by default but can be changed to UTC if need be.
HTTP info "GET /zato/ping HTTP/1.1" HTTP method, URL path invoked and HTTP method
Status code 200 HTTP status code returned in response
Response size 141 Size of the response in bytes
Referer "-" HTTP referer (sic), currently not used
User agent "curl/7.22.0 (x86_64-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3" User agent issuing the request

A feature that has recently landed in git master on GitHub and will be released in 2.0 is the ability to log into Zato's web-admin using OpenID.

This lets one make use of an already existing Single Sign-On (SSO) infrastructure instead of requiring Zato admins to memorize additional credentials.

Here's how to enable it:

  • Open the config file at /path/to/web/admin/config/repo/web-admin.conf

  • Change the OPENID_SSO_SERVER_URL to a URL your SSO server uses

  • Stop and start the web admin

  • For each user in web admin:

    • Make sure the user has been already created, let's say it's 'myuser'
    • Issue the new zato update openid command, for instance % zato update openid /path/to/web/admin myuser https:// sso.example.com/myuser OK %

Where

  • zato update openid - the command to invoke
  • /path/to/web/admin - path to web admin's top-level directory
  • myuser - username whose OpenID claimed ID should be set https://sso.example.com/myuser - claimed ID of the user

No restarts are needed after updating a given user's credentials.

Note that enabling SSO disables regular password based authentication. To revert to the latter, set OPENID_SSO_SERVER_URL to "" and restart web admin.

OVERVIEW

It has been possible to create reusable REST services in Zato right from the release 1.0 however a feature that recently landed in GitHub and that will be released in Zato 2.0 makes it easier to use of RESTful URLs, also known as clean URLs.

This blog post presents HTTP channels and an upcoming one will deal with HTTP outgoing connections.

THE GOAL

The diagram below depicts what we'd like to achieve - 3 independent applications invoking the same service yet each using separate URLs. To keep it simple, let's say we want to return a customer's order in JSON.

  • App1 uses GET and URL in format of /app1/customer/{cid}/order/{oid}
  • App2 uses POST to send JSON with customer ID while the order ID is in the URL /app2/order_id{oid}
  • App3 uses GET and sends parameters in query string, e.g. /app3?cid=111&oid=222 Each of them should invoke the same reusable service that should receive parameters on input without any changes to the code regardless of which application invokes it.

A diagram

SERVICE CODE

This is how the service we're exposing will look like. Note that for clarity, the same response is always produced though input data is copied over so we can quickly confirm on command line they were indeed received by the service.

Also note that we make use of SimpleIO - a declarative syntax for expressing I/O parameters that lets one use Python dictionaries, bunches or SQLAlchemy objects directly in order to forget about manual serialization - no matter if it's JSON or XML/SOAP, though only the former is used in this post.

# Zato
from zato.server.service import Service

class GetOrder(Service):
    """ Returns details of a given order.
    """
    class SimpleIO:
        input_required = ('cid', 'oid')
        output_required = ('cid', 'oid', 'value', 'date')

    def handle(self):

        # Copy over from request
        self.response.payload.cid = self.request.input.cid
        self.response.payload.oid = self.request.input.oid

        # Dummy order details
        self.response.payload.value = 11223344
        self.response.payload.date = '2013-10-27'

        # An appropriate content type for the payload returned
        self.response.content_type = 'application/x-order-details'

The service can be hot-deployed in the cluster and we can expose it through channels.

CHANNELS

3 separate HTTP channels for each of the applications.

Screenshot

Screenshot

Screenshot

INVOKING FROM COMMAND LINE

Having deployed the service and created the channels, we can proceed to invoking the services. Note that no restarts are required.

JSON output has been slightly reformatted for clarity.

APP1

$ curl localhost:11223/app1/customer/11/order/22
{
  "response": {
    "cid": "11",
    "oid": "22",
    "value": 11223344,
    "date": "2013-10-27"
  }
}

APP2

$ curl localhost:11223/app2/order_id44 -d '{"cid":"33"}'
{
  "response": {
    "cid": "33",
    "oid": "44",
    "value": 11223344,
    "date": "2013-10-27"
  }
}

APP3

$ curl "localhost:11223/app3?cid=55&oid=66"
{
  "response": {
    "cid": "55",
    "oid": "66",
    "value": 11223344,
    "date": "2013-10-27"
  }
}

A NOTE ON INVOKING

We couldn't use the built-in service invoker GUI as this bypasses channels altogether and while it is in fact a great help in development, it can't really be used here when the gist of the matter are channels themselves.

TWEAKING

How URL parameters are handled is governed by 3 options each channel offers:

Option Notes
Merge URL params to req Whether parameters found in URL, no matter if path or query string, should be included in sel
URL params priority What to do if parameters in path conflict with query string ones, i.e. what wins if both are provided, such as in /customer/123/?cid=456
Params priority After obtaining parameters from URL (taking their relative priority into account), how to resolve conflicts if both the URL and request body contain the same parameter, which one has higher priority

CONCLUSION

RESTful channels expand means through which reusable Zato-based services can be exposed to external applications. The feature is slated for inclusion in 2.0 and is already available from GitHub, in the master branch.