Imagine, or recall, a scenario. You integrate three applications, two client HTTP ones and the third is a backend one (no matter the technology).
All is well except client1 tends to send requests in bursts for no good reason and its developers just cannot tame it.
The backend app can't deal with it, they prefer a steady inflow of messages, and you're tasked with containing the issue.
In that situation you can take advantage of the fact that Zato uses an embedded instance of HAProxy as its load-balancer through which HTTP apps connect so you can fire up the GUI in config's source code view and add these lines to 'frontend front_http_plain'
so the whole of if reads now:
The GUI can be now used to validate and save the config:
There's a couple of assumptions
Instead of HAProxy 1.4, you use 1.5 (or later, but this is the latest version as of the time of this writing)
Each client accesses URLs beginning with a client-specific prefix - but this is a good
It was always possible to use GUI, API or a service to ping an outgoingHTTP/SOAP connection - the idea behind it is that you pretty much always want to check whether it is your servers that can connect to remote services not that you can access remote resources from your local host as it doesn't tell you, for instance, if firewalls between the servers and outside world are configured properly.
As suggested on GitHub, a new thing in 2.0 will be the option to specify what HTTP method to use for pings. This used to be always HEAD but can now be configured though HEAD is still the default one if you don't choose anything else.
Did you know that you could invoke any service from command line? This works with both Zato's own API and the services you develop. Depending on how a service is implemented/what it's exposed through, it can be done with JSON, plain XML, SOAP or any other data format a service accepts.
This will work even if a service isn't available over any channel and can be used to quickly test things out or to script Zato in bash or any other shell language.
For instance, browsing the API we can see there's a zato.security.basic-auth.get-list service which returns a list of all HTTP Basic Auth security definitions on a given cluster. Given that it's part of the API it can be invoked from command line using JSON or XML (or SOAP but let's skip that part here), as below. Output a bit reformatted for clarity
This also works for asynchronous invocations and can be used to make a service think it's been invoked through any channel type.
Summary: The post introduces Zato, an open-source integration platform in Python, and shows you how to integrate Django, or indeed any piece of Python software, with Zato and external web services using nothing but plain Python objects.
Applications in any programming language can be integrated using Zato but being written in Python itself, Zato offer a convenience client for software in Python and that will be used throughout the text.
Zato is a lightweight, yet complete, ESB (Enterprise Service Bus). And the project's goal is to become a powerful, yet lightweight, one.
Start here for a gentle introduction to what ESB and SOA (Service-Oriented Architecture) are about, but in short, they let you integrate multiple applications each potentially using different formats, protocols and programming languages with the aim of supporting interesting processes you need to automate. And with Zato this is all in pure Python with as little headaches as possible.
HOW THINGS SHOULD STAND
As a Python programmer, about the only thing I feel I should need in order to invoke web services exposed by any sort of systems is a simple API based on dicts or other dict-like objects, like Bunch.
It should be always possible to write code like what is below and expect it will just work regardless of the complexity of underlying protocols and data transports.
Given that it's a blog of the Zato project it won't come as a surprise that I am about to tell you that Zato allows you to achieve just that, to think in terms of services and dictionaries without having to worry about how everything is actually implemented underneath.
You delegate the job of an actual integration to Zato which becomes the component responsible for dealing with protocols and data formats, fetching information, straightening it and returning to you a unified view. This lets you focus on your job only and nicely follows the UNIX philosophy of separating software into clearly defined blocks interoperating in order to achieve an interesting result. Not to mention that this what the integrations industry has been using to tackle such scenarios for decades now.
This way you can focus on your own app, not on data integration. Someone else takes care of it.
THE OVERALL SCHEME
The diagram depicts what we will achieve:
Users enters a currency code to find EUR exchange rates to in an HTML form
A Django application invokes a Zato client providing a Python dictionary with currencies selected on input.
Behind the scenes, the dictionary is converted into an HTTP JSON call but this is completely transparent to you as a Django programmer.
Zato receives the call already converted to a Bunch instance and invokes 3 web services provided by:
Yahoo! Finance CSV API
Google Calculator Pseudo-JSON API
European Central Bank XML API
Output from 3 different sources is converted to a clean Pythonic response sent back to Django
Django app receives a list of dictionaries on output ready to use in a template which is shown to the user
First, clone this repository (we'll call the directory you'll clone it to DJANGO_APP_DIR) and run DJANGO_APP_DIR/install.sh - this will use install or upgrade distribute and virtualenv and use pip/buildout to download a couple of dependencies and install everything under virtualenv.
You can now go to http://127.0.0.1:8188 and witness an '[Errno 111] Connection refused' error. This is OK. Zato is not running yet.
What you can already have a look though is the Django code. Basically, a middleware class is used to inject a Zato client and the client is used to invoke a service which will be defined in the next steps.
Let's see, this is how the middleware looks like..
And here's the view..
If it were a project where you'd be doing Django programming only then you could congratulate yourself. The code shown above is everything you need to write to invoke a Zato service and fetch the exchange rates.
This is 10 lines of Python code, counting imports or class definitions in. Without the boilerplate, it will be 2 or 3 lines of code needed to invoke web services.
OK, there's also a trivial piece of HTML, the gist of which is here ..
but that's it. There is nothing else on Django side, job well done!
$ cp exchangerates.py ~/tmp/qs-1/server1/pickup-dir/
Both servers will now confirm the deployment, each in its own log (~/tmp/qs-1/server1/logs/server.log):
INFO - Uploaded package id:, payload_name:[exchangerates.py]
The service is there but it can't be used yet.
The way Zato is designed, unless you insist on it your services will never need to directly deal with any addresses, they only need to fetch a connection by its name ('Yahoo Finance', 'Google Calculator' and 'European Central Bank') and its Zato's job to manage it. You only need to think about overall processes and I/O, not about where an external service to invoke is located. If the location ever changes, you'll update it using GUI, CLI or API and servers will pick up changes automatically, without any restarts.
Another point to make is that with Zato your code never exposes your own services over any specific transport (HTTP, AMQP and so on). This is also done via GUI, CLIorAPI.
In fact, if you're using SimpleIO (SIO), the very same service can be exposed over HTTP/AMQP/JMS WebSphere MQ/ZeroMQ with JSON, XML or SOAP (and CSV is coming soon) without any code changes at all. That depends on what the service does, if it's a synchronous or asynchronous one but that's the principle.
Also note that most of the abstractions Zato uses are usually convenience wrappers around best Python libraries out there.
For instance, you can use Python dicts but you can also always use the underlying requests library directly for HTTP calls - you're never forced to use what Zato believes will be enough for you, there's nothing preventing you from customizing things to your liking with tools Zato doesn't offer out of the box.
Likewise, say Zato doesn't have something by default, like SMTP connections. Given that you're using Python you can still send out emails in 5 lines of code. (And by the way, SMTP will be added to Zato soon so this will become 1 line of code).
Let's fill out a couple of forms in Zato's GUI to make all the resources need by the service available. Note that it all can be done in JSON and stored in a config repository of your liking but let's use a GUI here.
You don't need to restart server after creating any object.
HTTP BASIC AUTH DEFINITION
Create a new definition and update its password to 'django-password' after it's created - by default passwords are set to randomly generated UUID4s (there are no default passwords in Zato at all).
PLAIN HTTP CHANNEL
Create a new channel object and assign a newly created security definition to it. Note that this particular Python client requires the service to be 'zato.service.invoke' and this is the service that invokes the one of yours.
OUTGOING PLAIN HTTP CONNECTIONS
An outgoing connection encapsulates information that is to do with particularities of a given transport method. This is everything that a service shouldn't be concerned with in its own code, such as endpoints, queues, URLs, authentication and so on. Zato deals with it itself, you just need to focus on your own functionality.
EUROPEAN CENTRAL BANK
RUNNING IT ALL
Now that everything has been created you can visit the Django app at http://127.0.0.1:8188/ and play around with various currencies - this will fetch everything from backend web services and display it in an HTML table.
WHAT ELSE IS THERE?
Naturally, this isn't everything. If you've already read the intro to ESB/SOA, you know the first question will be, is the service IRA?
Sure, if you need exchange rates in your projects such information will be certainly interesting on more than one occasion
Almost, the list of providers is hard-coded but ultimately, there should be one or more default provider and client applications should be able to specify which ones they're interested in
Yes, as long as it will be given the feature mentioned above (default providers, client apps say which one to use)
It also makes sense to use Zato's built-in scheduler and Redis to pre-fetch the rates periodically instead of accessing remote resources for each client request.
The good news is, such things are trivial to add with Zato and once you complete it, you'll have a truly IRA service that can be reused across a wide range of projects without any code changes. And your client apps will be always able to use plain dicts only.
CAN ZATO DO MORE?
There's a whole lot more Zato can do - JSON, SOAP, AMQP, JMS WebSphere MQ, ZeroMQ, Redis, SQL, FTP, load-balancing, scheduling, statistics, hooks, GUI, CLI, API - the features are there.
Note that Django was used in the text but the client is completely framework-agnostic, the same code will work with any Python application.
Also, Zato is in Python but it's not for integrating Python apps only. As long as your application can speak any of the protocols mentioned (this is 99% of apps out there), you're good to go.