Python libraries in a separate interpreter

Run a library in the interpreter it requires and call it from services as if it were local.

Run a Python library in an interpreter of its own when it requires a different Python version, when it conflicts with the server's dependencies or when it misbehaves in a shared process. The Connector SDK runs the library in the interpreter you point it at - any Python on the machine - and your connector calls it as if it were local.

The hosting side ships with Zato - a stock runner module that any interpreter runs by path - and the module you write for it is plain Python.

The module with your library

The module contains plain module-level functions and imports only the library it wraps:

# -*- coding: utf-8 -*-

# Anything the library needs
import the_library_that_cannot_run_in_the_server

def ping():
    return 'pong'

def transform(text):
    return the_library_that_cannot_run_in_the_server.process(text)

The connector module

The connector starts the library's interpreter with start_process and calls it through RunnerClient - each call names a function of your module:

# -*- coding: utf-8 -*-

# stdlib
import time

# Zato
from zato.common.sdk import Connector, Field
from zato.common.sdk import runner
from zato.common.sdk.runner import RunnerClient

# How long to wait for the runner process to start accepting connections, in seconds.
_startup_timeout = 15

class TextProcConnector(Connector):
    """ Wraps a Python library that runs in a separate interpreter - the stock runner hosts
    the library's module and this connector calls it over a local socket.
    """
    type = 'textproc'

    # Configuration schema - which interpreter runs the runner and which module the runner exposes.
    python_path = Field.Text()
    module_path = Field.Text()

    def create_client(self) -> 'RunnerClient':

        # Run the stock runner as a supervised helper process, in a clean interpreter -
        # the runner depends on the standard library only, so any interpreter can run it by path.
        command = [self.config.python_path, runner.__file__, '{port}', self.config.module_path]
        process = self.start_process(command)

        client = RunnerClient('127.0.0.1', process.port)

        # Wait until the runner accepts connections.
        deadline = time.monotonic() + _startup_timeout

        while True:
            try:
                client.call('ping')
            except OSError:
                if time.monotonic() > deadline:
                    raise Exception(f'The runner did not start within {_startup_timeout}s')
                time.sleep(0.2)
            else:
                break

        return client

    def ping(self, client:'RunnerClient') -> 'None':
        client.call('ping')

    def transform(self, text:'str') -> 'str':
        return self.client.call('transform', text)

python_path selects the interpreter that runs the library - any Python on the machine. The runner depends only on the standard library, so a bare interpreter is enough.

The platform supervises the library's process like any helper started with start_process. When the process exits, the platform rebuilds the connection, which starts a new one.

RunnerClient.call sends one JSON object per line and reads one back. An exception raised by your module arrives on the connector's side as an exception.

Create a definition

Create the definition in an enmasse file, under the custom_textproc key - custom_ followed by the connector's type:

custom_textproc:
  - name: My TextProc
    python_path: /opt/python312/bin/python
    module_path: /opt/company/textproc_module.py

The library runs in the interpreter the definition names and services call it through the named connection.

Page	What it covers
SDK reference	Field types, lifecycle methods and the behavior of every invocation
Java, .NET and native code	Vendor components run as supervised helper processes
Connector SDK tutorial	Build a complete connector and call it from a service

Python libraries in a separate interpreter

The module with your library

The connector module

Create a definition

See also