This GitHub repository contains Zato code developed during WSGI Wrestle 2013.

INTRO

It's a middleware cache for exchange rates data fetched using Yahoo! YQL's XML. It's meant to support hypothetical, yet practical, scenarios of storing backend data in a cache close to a frontend application. The client frontend is not included in the repository because Zato is an ESB and backend application server, not a frontend one.

The code can form part of a wider process of building analytical applications that need convenient access to aggregated data.

Splitting the functionality across multiple applications means higher reusability - Zato provides caching, API and data while frontend systems - no matter what technology they are written with - can focus on their own job of providing interesting and useful user interfaces.

A diagram

WHAT IT SUPPORTS AND DOES

  • Querying for an average exchange rate for a given date
  • Registering pairs of currency codes to periodically grab rates for (i.e. USD and EUR)
  • Deleting existing pairs
  • Listing all existing pairs
  • Periodically updating the cache with newest values read using YQL's XML response
  • Periodically trimming the cache so it doesn't grow indefinitely

All that is contained within a single Python module hot-deployed on servers running in a cluster behind a high-availability load-balancer.

SAMPLE USAGE

The installation documentation includes exposing two of the services over HTTP and JSON, so once everything is installed ...

Screenshot Screenshot

... curl can be used to register a couple of pairs and querying for their current exchange rate.

400: Invalid request

NOTES

  • Zato servers run in clusters that are built using gevent/gunicorn with HAProxy in front of them. This is completely transparent from a developer's viewpoint.

  • Services hot-deployed on one server in a cluster are automatically installed on other servers.

  • There are several services involved and each can be independently exposed over multiple access channels though the installation steps show only create-exchange-pair and get-rate.

  • update-cache and trim-cache modify the contents of the cache with a Redis-based distributed lock held. This is needed because each service may be possibly running in multiple copies on more than one server so there must be a way to ensure that each one has access to consistent data only - no partial updates must be visible.

  • dispatch-update-cache is a wrapper service which uses self.invoke_async to asynchronously invoke a separate instance of update-cache which actually updates the cache. Each instance may be running on different server and the load is spread automatically by Zato.

  • The solution developed can invoked from any frontend application using any programming language but for Python applications, Zato offers a Python client which allows one to communicate with Zato using regular Python dictionaries, as below

400: Invalid request

The client is built on top of the requests library and can be invoked from Django, Flask or any other Python code.