This GitHub repository contains Zato code developed during WSGI Wrestle 2013.

INTRO

It's a middleware cache for exchange rates data fetched using Yahoo! YQL's XML. It's meant to support hypothetical, yet practical, scenarios of storing backend data in a cache close to a frontend application. The client frontend is not included in the repository because Zato is an ESB and backend application server, not a frontend one.

The code can form part of a wider process of building analytical applications that need convenient access to aggregated data.

Splitting the functionality across multiple applications means higher reusability - Zato provides caching, API and data while frontend systems - no matter what technology they are written with - can focus on their own job of providing interesting and useful user interfaces.

A diagram

WHAT IT SUPPORTS AND DOES

  • Querying for an average exchange rate for a given date
  • Registering pairs of currency codes to periodically grab rates for (i.e. USD and EUR)
  • Deleting existing pairs
  • Listing all existing pairs
  • Periodically updating the cache with newest values read using YQL's XML response
  • Periodically trimming the cache so it doesn't grow indefinitely

All that is contained within a single Python module hot-deployed on servers running in a cluster behind a high-availability load-balancer.

SAMPLE USAGE

The installation documentation includes exposing two of the services over HTTP and JSON, so once everything is installed ...

Screenshot Screenshot

... curl can be used to register a couple of pairs and querying for their current exchange rate.

NOTES

  • Zato servers run in clusters that are built using gevent/gunicorn with HAProxy in front of them. This is completely transparent from a developer's viewpoint.

  • Services hot-deployed on one server in a cluster are automatically installed on other servers.

  • There are several services involved and each can be independently exposed over multiple access channels though the installation steps show only create-exchange-pair and get-rate.

  • update-cache and trim-cache modify the contents of the cache with a Redis-based distributed lock held. This is needed because each service may be possibly running in multiple copies on more than one server so there must be a way to ensure that each one has access to consistent data only - no partial updates must be visible.

  • dispatch-update-cache is a wrapper service which uses self.invoke_async to asynchronously invoke a separate instance of update-cache which actually updates the cache. Each instance may be running on different server and the load is spread automatically by Zato.

  • The solution developed can invoked from any frontend application using any programming language but for Python applications, Zato offers a Python client which allows one to communicate with Zato using regular Python dictionaries, as below

The client is built on top of the requests library and can be invoked from Django, Flask or any other Python code.