self.commands.invoke_async to execute shell commands and command-line programs. Commands can run in foreground,
which means that your service will wait for their output, or they can run in background, which does not block
This makes it possible to design API services and workflows whose core logic involves or revolves around the usage of command-line tools - accepting data from external sources, processing them using tools invoked as though they ran from the command line and notifying external systems or applications when results are ready.
Several options can be provided to configure the details of how to execute the commands, e.g. what timeouts to use or what callbacks to invoke when the command completes.
# -*- coding: utf-8 -*- # Zato from zato.server.service import Service class MyService(Service): def handle(self): # Command to execute .. command = 'cd /tmp && ls -la && whoami && uname -a' # .. invoke it now .. result = self.commands.invoke(command) # .. log the data returned .. self.logger.info('Stdout -> %s', result.stdout) self.logger.info('Stderr -> %s', result.stderr) # .. log exit code and other useful information as well. self.logger.info('Exit code -> %s', result.exit_code) self.logger.info('Timed out -> %s', result.is_timeout) # Consult the documentation below for more usage examples
Each command invocation produces a
CommandResult object that encapsulates information about the results produced
by the command and other useful metadata, e.g. it will contain the exit code, stdout, stderr or the total time the command
took to complete.
|is_timeout||bool||False||True if the command
|timeout_msg||str||---||If is_timeout is True, a human-readable message about the fact that
|total_time||str||--||As above, as a string, e.g. '0:00:01.35'.|
|command||str||---||The body of the command that was executed.|
|is_ok||bool||---||True if exit code is 0 or -1, False otherwise.|
|cid||str||---||Correlation ID assigned to the command. If you do not provide your own, one will be auto-generated.|
|callback||---||---||A callback that was invoked when the command completed. It could be a regular Python function or method, a service or a pub/sub topic. Described fuller later in the chapter.|
|stdin||str||---||If the command required for stdin to be provided, what it was.|
|is_async||bool||False||True if the command ran in background, False otherwise.|
|use_pubsub||str||False||True if the result object was delivered to the callback using a pub/sub topic.|
|len_stdout_bytes||int||-1||How many bytes of stdout the command produced.|
|len_stderr_bytes||int||-1||How many bytes of stderr the command produced.|
|len_stdout_human||str||---||A human-readable representation of len_stdout_bytes, e.g. '146.1 kB' instead of 146102|
|len_stderr_human||str||---||As above, for stderr.|
|encoding||str||utf8||What encoding was used to convert bytes that the command produced to a string object.|
|replace_char||str||� (U+FFFD)||What character was applied to replace stdout bytes that could not be used by the encoding specified. For instance, you can use the "?" character instead of the default one to keep everything encoded in ASCII. The default value is U+FFFD REPLACEMENT CHARACTER, per the Unicode specification.|
|start_time||datetime||---||When the command started (in UTC).|
|start_time_iso||str||---||As above, as a string.|
|end_time||datetime||---||When the command completed (in UTC).|
|end_time_iso||str||---||As above, as a string.|
self.commands.invoke_async to invoke a command asynchronously, in background. You can specify an optional
callback to be invoked when the command finishes. More about callbacks later in the chapter.
# -*- coding: utf-8 -*- # Zato from zato.server.commands import CommandResult from zato.server.service import Service class MyService(Service): def on_completed(self, result:CommandResult) -> None: # This will run when the command has finished self.logger.info('Received result: %s', result) def handle(self): # A command that will take a longer time to complete .. command = 'sleep 5' # .. invoke it in background, specifying a callback. self.commands.invoke_async(command, callback=self.on_completed)
Commands can span multiple lines. Use
&& - the regular Bash line-continuation syntax - to split a single line into many.
# -*- coding: utf-8 -*- # Zato from zato.server.service import Service class MyService(Service): def handle(self): # A multi-line command to execute .. command = """ cd ~ && ls -la && cd /tmp && ls -la """ # .. invoke it now .. result = self.commands.invoke(command) # .. log the result for inspection. self.logger.info('Result: %s', result)
When invoking a command, regardless of whether it runs in foreground or background, the following options can be specified using keyword arguments:
|timeout||float||600.0||In seconds, how long to wait for the command to complete. Use None to wait indefinitely.|
|callback||any||---||What callback to invoke when the command finishes. It can be provided both if the invocation is in foreground or background. Read below for a detailed description of callbacks.|
|use_pubsub||bool||False||Whether callbacks should be invoked by publishing messages to their publish/subscribe topics.|
|cid||str||---||Correlation ID. An arbitrary string that can be used to correlate the execution of the command with other events in the system, e.g. the same CID can be used elsewhere for identifying the REST request that accepted the command's input, in this way leading to a chain of events, all of them correlated using this ID. A random value will be generated if one is not given on input.|
|stdin||str||---||Some commands may require you to send stdin to them and this is how it can be provided.|
|encoding||str||utf8||What encoding to use for the command and its output.|
|replace_char||str||� (U+FFFD)||If stdout or stderr cannot be decoded into the encoding given on input, what character to use for the characters that could not be decoded. The default value of
Sample code using several invocation options:
A callback is a Python function, method, a Zato service or a publish/subscribe topic that is notified about the result of the command's execution.
Callbacks can be used to form arbitrary series of information exchanges, e.g. it is possible to run a command, have its stdout published to a REST endpoint, an AMQP queue and a topic. In turn, all subscribers of messages from that topic will receive their own notifications. Other types of endpoints can be used freely.
Callbacks can be specified for both foreground and background commands. If it is the former, the callback runs before the service executing the command receives the command's result. If it is the latter, the invoking service receives a basic results object that specifies the invocation's correlation ID and other metadata. The command then runs in background and the callback is invoked when the command exists.
With commands that run in background, it is possible that the callback will run before the service that runs the command receives the results object - this is possible because the service and callback run independently, as separate tasks, and, if the command exits very quickly, its callback will execute first.
# -*- coding: utf-8 -*- # Zato from zato.server.commands import CommandResult from zato.server.service import Service # App from myapi import MyCallbackService class MyService(Service): def on_completed(self, result:CommandResult) -> None: # This will run when the command has finished self.logger.info('Received result: %s', result) def handle(self): # A command that will take a longer time to complete .. command = 'sleep 5' # Invokes the command in background and then a callback method is invoked self.commands.invoke_async(command, callback=self.on_completed) # Invokes the command in foreground and then invokes another service by its name self.commands.invoke(command, callback='my.callback.service') # Invokes the command in foreground and then invokes a previously imported service self.commands.invoke(command, callback=MyCallbackService) # These two calls invoke a command and then use publish/subscribe # to invoke its callbacks. The first command runs in foreground, # the other in background. self.commands.invoke(command, callback=MyCallbackService, use_pubsub=True) self.commands.invoke_async(command, callback='my.callback.service', use_pubsub=False) # This background invocation publishes a message to a named pub/sub topic as its callback self.commands.invoke_async(command, callback='/commands/results', use_pubsub=True)
All commands execute in a shell of the same operating system user that a given Zato server runs under, typically, it is
It means that all the permissions, environment variables or other shell constructs apply to each command as well.
It means that commands can do everything that the user can do, for instance, if the user has specific SSH keys, the commands can use them,
if the user is allowed to access
/etc, the commands executed by services will also be allowed to access that directory.
It also means that you must ensure that commands use only sanitized and vetted data from trusted sources only. As an example,
if there is a command called 'myprogram' that accepts an option called '--email', you need to ensure that the value provided
for this option truly is an email as otherwise, if sanitization were insufficient, an attacker could use a value such as
'firstname.lastname@example.org ; rm -rf ~', which would result in the command of
myprogram --email email@example.com ; rm -rf ~,
which would first invoke the command and then delete the user's home directory.
Note also that when you log the results of the command's execution, the information saved to logs will contain all the details about the command, including its stdin, stdout and stderr. Some of this information may be sensitive - make sure that you log only the very specific information that you need.
For instance, during the initial development it is useful to simply use this:
.. but as soon as possible you may want to log only more specific details, e.g.: