Python shell commands

Use self.commands.invoke or self.commands.invoke_async to execute shell commands and command-line programs. Commands can run in foreground, which means that your service will wait for their output, or they can run in background, which does not block your service.

This makes it possible to design API services and workflows whose core logic involves or revolves around the usage of command-line tools - accepting data from external sources, processing them using tools invoked as though they ran from the command line and notifying external systems or applications when results are ready.

Several options can be provided to configure the details of how to execute the commands, e.g. what timeouts to use or what callbacks to invoke when the command completes.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class MyService(Service):

    def handle(self):

        # Command to execute ..
        command = 'cd /tmp && ls -la && whoami && uname -a'

        # .. invoke it now ..
        result = self.commands.invoke(command)

        # .. log the data returned ..
        self.logger.info('Stdout -> %s', result.stdout)
        self.logger.info('Stderr -> %s', result.stderr)

        # .. log exit code and other useful information as well.
        self.logger.info('Exit code -> %s', result.exit_code)
        self.logger.info('Timed out -> %s', result.is_timeout)

        # Consult the documentation below for more usage examples

Result objects

Each command invocation produces a CommandResult object that encapsulates information about the results produced by the command and other useful metadata, e.g. it will contain the exit code, stdout, stderr or the total time the command took to complete.

AttributeDatatypeDefault valueNotes
stdoutstr---Stdout produced by the command.
stderrstr---Stderr produced by the command.
exit_codeint-1Exit code that the command returned. A value of -1 means that the exit code is unknown, e.g. the command timed out.
timeoutfloat600.0Timeout in seconds used for the command's execution. Defaults to 10 minutes.
is_timeoutboolFalseTrue if the command timed out.
timeout_msgstr---If is_timeout is True, a human-readable message about the fact that a timeout has occurred.
total_time_secfloat---Total time in seconds the command took to complete, e.g. 1.35.
total_timestr--As above, as a string, e.g. '0:00:01.35'.
commandstr---The body of the command that was executed.
is_okbool---True if exit code is 0 or -1, False otherwise.
cidstr---Correlation ID assigned to the command. If you do not provide your own, one will be auto-generated.
callback------A callback that was invoked when the command completed. It could be a regular Python function or method, a service or a pub/sub topic. Described fuller later in the chapter.
stdinstr---If the command required for stdin to be provided, what it was.
is_asyncboolFalseTrue if the command ran in background, False otherwise.
use_pubsubstrFalseTrue if the result object was delivered to the callback using a pub/sub topic.
len_stdout_bytesint-1How many bytes of stdout the command produced.
len_stderr_bytesint-1How many bytes of stderr the command produced.
len_stdout_humanstr---A human-readable representation of len_stdout_bytes, e.g. '146.1 kB' instead of 146102
len_stderr_humanstr---As above, for stderr.
encodingstrutf8What encoding was used to convert bytes that the command produced to a string object.
replace_charstr� (U+FFFD)What character was applied to replace stdout bytes that could not be used by the encoding specified. For instance, you can use the "?" character instead of the default one to keep everything encoded in ASCII. The default value is U+FFFD REPLACEMENT CHARACTER, per the Unicode specification.
start_timedatetime---When the command started (in UTC).
start_time_isostr---As above, as a string.
end_timedatetime---When the command completed (in UTC).
end_time_isostr---As above, as a string.

Running commands in background

Use self.commands.invoke_async to invoke a command asynchronously, in background. You can specify an optional callback to be invoked when the command finishes. More about callbacks later in the chapter.

# -*- coding: utf-8 -*-

# Zato
from zato.server.commands import CommandResult
from zato.server.service import Service

class MyService(Service):

    def on_completed(self, result:CommandResult) -> None:

        # This will run when the command has finished
        self.logger.info('Received result: %s', result)

    def handle(self):

        # A command that will take a longer time to complete ..
        command = 'sleep 5'

        # .. invoke it in background, specifying a callback.
        self.commands.invoke_async(command, callback=self.on_completed)

Multi-line commands

Commands can span multiple lines. Use && - the regular Bash line-continuation syntax - to split a single line into many.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class MyService(Service):

    def handle(self):

        # A multi-line command to execute ..
        command = """
        cd ~    &&
        ls -la  &&
        cd /tmp &&
        ls -la
        """

        # .. invoke it now ..
        result = self.commands.invoke(command)

        # .. log the result for inspection.
        self.logger.info('Result: %s', result)

Invocation options

When invoking a command, regardless of whether it runs in foreground or background, the following options can be specified using keyword arguments:

AttributeDatatypeDefault valueNotes
timeoutfloat600.0In seconds, how long to wait for the command to complete. Use None to wait indefinitely.
callbackany---What callback to invoke when the command finishes. It can be provided both if the invocation is in foreground or background. Read below for a detailed description of callbacks.
use_pubsubboolFalseWhether callbacks should be invoked by publishing messages to their publish/subscribe topics.
cidstr---Correlation ID. An arbitrary string that can be used to correlate the execution of the command with other events in the system, e.g. the same CID can be used elsewhere for identifying the REST request that accepted the command's input, in this way leading to a chain of events, all of them correlated using this ID. A random value will be generated if one is not given on input.
stdinstr---Some commands may require you to send stdin to them and this is how it can be provided.
encodingstrutf8What encoding to use for the command and its output.
replace_charstr� (U+FFFD)If stdout or stderr cannot be decoded into the encoding given on input, what character to use for the characters that could not be decoded. The default value of is what the Unicode specification uses but it may be overridden in this way in case another character is more suitable.

Sample code using several invocation options:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class MyService(Service):

    def handle(self):

        # Command to execute ..
        command = 'ls -la /tmp'

        # .. invoke it with a few options.
        self.commands.invoke(command, cid='abc123', timeout=90, encoding='iso-8859-1')

Using callbacks

A callback is a Python function, method, a Zato service or a publish/subscribe topic that is notified about the result of the command's execution.

Callbacks can be used to form arbitrary series of information exchanges, e.g. it is possible to run a command, have its stdout published to a REST endpoint, an AMQP queue and a topic. In turn, all subscribers of messages from that topic will receive their own notifications. Other types of endpoints can be used freely.

Callbacks can be specified for both foreground and background commands. If it is the former, the callback runs before the service executing the command receives the command's result. If it is the latter, the invoking service receives a basic results object that specifies the invocation's correlation ID and other metadata. The command then runs in background and the callback is invoked when the command completes.

With commands that run in background, it is possible that the callback will run before the service that runs the command receives the results object - this is possible because the service and callback run independently, as separate tasks, and, if the command exits very quickly, its callback will execute first.

# -*- coding: utf-8 -*-

# Zato
from zato.server.commands import CommandResult
from zato.server.service import Service

# App
from myapi import MyCallbackService

class MyService(Service):

    def on_completed(self, result:CommandResult) -> None:

        # This will run when the command has finished
        self.logger.info('Received result: %s', result)

    def handle(self):

        # A command that will take a longer time to complete ..
        command = 'sleep 5'

        # Invokes the command in background and then a callback method is invoked
        self.commands.invoke_async(command, callback=self.on_completed)

        # Invokes the command in foreground and then invokes another service by its name
        self.commands.invoke(command, callback='my.callback.service')

        # Invokes the command in foreground and then invokes a previously imported service
        self.commands.invoke(command, callback=MyCallbackService)

        # These two calls invoke a command and then use publish/subscribe
        # to invoke its callbacks. The first command runs in foreground,
        # the other in background.
        self.commands.invoke(command, callback=MyCallbackService, use_pubsub=True)
        self.commands.invoke_async(command, callback='my.callback.service', use_pubsub=False)

        # This background invocation publishes a message to a named pub/sub topic as its callback
        self.commands.invoke_async(command, callback='/commands/results', use_pubsub=True)

Security considerations

All commands execute in a shell of the same operating system user that a given Zato server runs under, typically, it is zato. It means that all the permissions, environment variables or other shell constructs apply to each command as well.

It means that commands can do everything that the user can do, for instance, if the user has specific SSH keys, the commands can use them, if the user is allowed to access /etc, the commands executed by services will also be allowed to access that directory.

It also means that you must ensure that commands use only sanitized and vetted data from trusted sources only. As an example, if there is a command called 'myprogram' that accepts an option called '--email', you need to ensure that the value provided for this option truly is an email as, otherwise, if sanitization were insufficient, an attacker could use a value such as 'user@example.com ; rm -rf ~', which would result in the command of myprogram --email user@example.com ; rm -rf ~, which would first invoke the command and then delete the user's home directory.

Note also that when you log the results of the command's execution, the information saved to logs will contain all the details about the command, including its stdin, stdout and stderr. Some of this information may be sensitive - make sure that you log only the very specific information that you need.

For instance, during the initial development it is useful to simply use this:

self.logger.info('Result: %s', result)

.. but as soon as possible you may want to log only more specific details, e.g.:

self.logger.info('CID: %s; Exit code: %s', result.cid, result.exit_code)


Schedule a meaningful demo

Book a demo with an expert who will help you build meaningful systems that match your ambitions

"For me, Zato Source is the only technology partner to help with operational improvements."

— John Adams
Program Manager of Channel Enablement at Keysight