Python shell commands

Use self.commands.invoke or self.commands.invoke_async to execute shell commands and command-line programs. Commands can run in foreground, which means that your service will wait for their output, or they can run in background, which does not block your service.

This makes it possible to design API services and workflows whose core logic involves or revolves around the usage of command-line tools - accepting data from external sources, processing them using tools invoked as though they ran from the command line and notifying external systems or applications when results are ready.

Several options can be provided to configure the details of how to execute the commands, e.g. what timeouts to use or what callbacks to invoke when the command completes.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class MyService(Service):

    def handle(self):

        # Command to execute ..
        command = 'cd /tmp && ls -la && whoami && uname -a'

        # .. invoke it now ..
        result = self.commands.invoke(command)

        # .. log the data returned ..
        self.logger.info('Stdout -> %s', result.stdout)
        self.logger.info('Stderr -> %s', result.stderr)

        # .. log exit code and other useful information as well.
        self.logger.info('Exit code -> %s', result.exit_code)
        self.logger.info('Timed out -> %s', result.is_timeout)

        # Consult the documentation below for more usage examples

Result objects

Each command invocation produces a CommandResult object that encapsulates information about the results produced by the command and other useful metadata, e.g. it will contain the exit code, stdout, stderr or the total time the command took to complete.

Running commands in background

Use self.commands.invoke_async to invoke a command asynchronously, in background. You can specify an optional callback to be invoked when the command finishes. More about callbacks later in the chapter.

# -*- coding: utf-8 -*-

# Zato
from zato.server.commands import CommandResult
from zato.server.service import Service

class MyService(Service):

    def on_completed(self, result:CommandResult) -> None:

        # This will run when the command has finished
        self.logger.info('Received result: %s', result)

    def handle(self):

        # A command that will take a longer time to complete ..
        command = 'sleep 5'

        # .. invoke it in background, specifying a callback.
        self.commands.invoke_async(command, callback=self.on_completed)

Multi-line commands

Commands can span multiple lines. Use && - the regular Bash line-continuation syntax - to split a single line into many.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class MyService(Service):

    def handle(self):

        # A multi-line command to execute ..
        command = """
        cd ~    &&
        ls -la  &&
        cd /tmp &&
        ls -la
        """

        # .. invoke it now ..
        result = self.commands.invoke(command)

        # .. log the result for inspection.
        self.logger.info('Result: %s', result)

Invocation options

When invoking a command, regardless of whether it runs in foreground or background, the following options can be specified using keyword arguments:

Sample code using several invocation options:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class MyService(Service):

    def handle(self):

        # Command to execute ..
        command = 'ls -la /tmp'

        # .. invoke it with a few options.
        self.commands.invoke(command, cid='abc123', timeout=90, encoding='iso-8859-1')

Using callbacks

A callback is a Python function, method, a Zato service or a publish/subscribe topic that is notified about the result of the command's execution.

Callbacks can be used to form arbitrary series of information exchanges, e.g. it is possible to run a command, have its stdout published to a REST endpoint, an AMQP queue and a topic. In turn, all subscribers of messages from that topic will receive their own notifications. Other types of endpoints can be used freely.

Callbacks can be specified for both foreground and background commands. If it is the former, the callback runs before the service executing the command receives the command's result. If it is the latter, the invoking service receives a basic results object that specifies the invocation's correlation ID and other metadata. The command then runs in background and the callback is invoked when the command completes.

With commands that run in background, it is possible that the callback will run before the service that runs the command receives the results object - this is possible because the service and callback run independently, as separate tasks, and, if the command exits very quickly, its callback will execute first.

# -*- coding: utf-8 -*-

# Zato
from zato.server.commands import CommandResult
from zato.server.service import Service

# App
from myapi import MyCallbackService

class MyService(Service):

    def on_completed(self, result:CommandResult) -> None:

        # This will run when the command has finished
        self.logger.info('Received result: %s', result)

    def handle(self):

        # A command that will take a longer time to complete ..
        command = 'sleep 5'

        # Invokes the command in background and then a callback method is invoked
        self.commands.invoke_async(command, callback=self.on_completed)

        # Invokes the command in foreground and then invokes another service by its name
        self.commands.invoke(command, callback='my.callback.service')

        # Invokes the command in foreground and then invokes a previously imported service
        self.commands.invoke(command, callback=MyCallbackService)

        # These two calls invoke a command and then use publish/subscribe
        # to invoke its callbacks. The first command runs in foreground,
        # the other in background.
        self.commands.invoke(command, callback=MyCallbackService, use_pubsub=True)
        self.commands.invoke_async(command, callback='my.callback.service', use_pubsub=False)

        # This background invocation publishes a message to a named pub/sub topic as its callback
        self.commands.invoke_async(command, callback='/commands/results', use_pubsub=True)

Security considerations

All commands execute in a shell of the same operating system user that a given Zato server runs under, typically, it is zato. It means that all the permissions, environment variables or other shell constructs apply to each command as well.

It means that commands can do everything that the user can do, for instance, if the user has specific SSH keys, the commands can use them, if the user is allowed to access /etc, the commands executed by services will also be allowed to access that directory.

It also means that you must ensure that commands use only sanitized and vetted data from trusted sources only. As an example, if there is a command called 'myprogram' that accepts an option called '--email', you need to ensure that the value provided for this option truly is an email as, otherwise, if sanitization were insufficient, an attacker could use a value such as 'user@example.com ; rm -rf ~', which would result in the command of myprogram --email user@example.com ; rm -rf ~, which would first invoke the command and then delete the user's home directory.

Note also that when you log the results of the command's execution, the information saved to logs will contain all the details about the command, including its stdin, stdout and stderr. Some of this information may be sensitive - make sure that you log only the very specific information that you need.

For instance, during the initial development it is useful to simply use this:

self.logger.info('Result: %s', result)

.. but as soon as possible you may want to log only more specific details, e.g.:

self.logger.info('CID: %s; Exit code: %s', result.cid, result.exit_code)