apitest - Zato API Testing

Introduction

zato-apitest is a command line tool for automated REST API testing. Tests are written in plain English, with no programming needed, and can be easily extended in Python if need be.

Here is how a built-in demo test case looks like:

zato-apitest demo run

What it can do

  • Invoke REST APIs
  • Use JSON Pointers to set request’s elements to strings, integers, floats, lists, random ones from a set of values, random strings, dates now/random/before/after/between.
  • Check that elements, exist, do not exist, that an element is an integer, float, list, empty, non-empty, that it belongs to a list or does not.
  • Set custom HTTP headers, user agent strings, method and SOAP action.
  • Check that HTTP headers are or are not of expected value, that a header exists or not, contains a value or not, is empty or not, starts with a value or not and ends with a value or not.
  • Read configuration from environment and config files.
  • Store values extracted out of previous steps for use in subsequent steps, i.e. get a list of objects, pick ID of the first one and use this ID in later steps.

Download and install

Newest releases are always available on PyPI and can be installed with pip, e.g.:

$ pip install zato-apitest

Run a demo test

Having installed the program, running apitest demo will set up a demo test case, run it against a live environment and present the results, as on the screenshot in Introduction above.

It is recommended that you study the demo closer, copy it over to a directory of your choice, and customise it to learn by experimenting with an actual set of assertions.

Workflow

  1. Install zato-apitest
  2. Initialize a test environment by running apitest init /path/to/an/empty/directory
  3. Add tests
  4. Execute apitest run /path/to/tests/directory to run tests
  5. Repeat tests 3 and 4 as often as needed

How does a test work?

Here is how a sample test kept in ./features/cust-update.feature may look like. This is the literal copy of a test, everything is in plain English:

Feature: Customer update

Scenario: REST customer update

    Given address "http://example.com"
    Given URL path "/json/customer"
    Given query string "?id=123"
    Given HTTP method "PUT"
    Given format "JSON"
    Given header "X-Server" "server-test-19"
    Given request "cust-update.json"
    Given path "/name" in request is "Maria"
    Given path "/last-seen" in request is UTC now "default"

    When the URL is invoked

    Then path "/action/code" is an integer "0"
    And path "/action/message" is "Ok, updated"
    And status is "200"
    And header "X-My-Header" is "My.Value"
  • Each test begins with a Feature: preamble which denotes what is being tested
  • Each scenario has 3 parts, corresponding to building a request, invoking an URL and running assertions on a response received:
  • One or more Given steps
  • Exactly one When step
  • One or more Then/And steps. There is no difference between how Then and And work, simply the first assertion is called Then and the rest of them is And. Any assertion may come first.
  • In both Given and Then/And the order of steps is always honored.
  • Steps work by matching patterns that can be potentially parametrized between double quotation marks, for instance Given address "http://example.com" is an invocation of a Given address "{address}" pattern.

Where to keep configuration

Configuration of the test scenarios can be kept in and read from 3 places:

  • Environment variables
  • ./features/config.ini
  • Test case-specifc context

The rules are:

  • Any value prefixed by ‘$’ is read from an environment variable
  • Any value prefixed by ‘@’ is read from ./features/config.ini’s [user] stanza
  • Any value prefixed by ‘#’ is read from the current test case’s context

Additionally, please keep in mind that individual tests can store variables basing on previous steps or responses hence combining all the configuration options allows one to form advanced scenarios, such as the one below.

Feature: zato-apitest docs

Scenario: Prepare data

    Given address "$MYAPP_ADDRESS"
    Given URL path "@MYAPP_PATH_LOGIN"
    Given format "JSON"
    Given I store "Maria Garca" under "cust_name"
    Given request is "{}"
    Given path "/customer_id" in request is "$MYAPP_DEFAULT_CUSTOMER"

    When the URL is invoked

    Then I store "/login" from response under "cust_login"

Scenario: Get customer payments

    Given address "$MYAPP_ADDRESS"
    Given URL path "@MYAPP_PATH_PAYMENTS"
    Given format "JSON"
    Given request is "{}"
    Given path "/cust_login" in request is "#cust_login"
    Given path "/cust_name" in request is "#cust_name"

    When the URL is invoked

    Then status is "200"

Discussion:

  • First scenario prepares data needed for the actual test performed by the second one
  • MYAPP_ADDRESS is an environment variable that can change from host to host without being hardcoded in test’s body
  • MYAPP_PATH_LOGIN is a variable stored in ./features/config.ini’s [user] stanza
  • Variable ‘cust_name’ is set to a static value of ‘Maria Garca’
  • Variable ‘cust_login’ is set to a value returned in response to the fist scenario
  • Second scenario makes use of data prepared by the first one
  • Remember to use clean up the current execution context using the “Then context is cleaned up” if you actually do not want for any config variables to be carried over from a step to subsequent ones

Note that when you give additional options to the apitest run command, they will be passed to the underlying test runner, just like the options stored in the config.ini file. For instance, in a test run you can exclude and include only certain features using the -e option and -i options.

Extending zato-apitest and adding custom assertions

It is easy to add new steps - let’s say that we need to add a step that will return the name of any weekday coming after one provided on input. So, for instance, if it is Thursday on input, the step should return Friday, Saturday or Sunday.

Each zato-apitest environment contains a ./features/steps/steps.py module. Initially, it only imports all the default steps but you can simply add your own steps to it. They are always based on behave so that is where all the additional details are explained.

Here is what needs to be added to ./features/steps/steps.py for the new step to be available:

from random import choice
from jsonpointer import set_pointer

week_days = 'Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun'

@given('path "{path}" in request is a weekday after "{start}"')
@obtain_values
def given_weekday_after(ctx, path, start):
    """ Returns any weekday after the start one.
    """
    # Sunday is the last day already
    if start == 'Sun':
        raise ValueError('{start} needs to be at most Sat')

    elif start not in week_days:
        raise ValueError('{{start}} ({}) needs to be among {}'.format(
            start, week_days))

    # Build a list of days to pick one from
    start_idx = week_days.index(start)
    remaining = week_days[start_idx+1:]

    # Pick any from the remaining ones after start
    value = choice(remaining)

    # Set it in request
    set_pointer(ctx, path, value)

Now you can make use of it in tests, for instance:

Feature: zato-apitest docs

Scenario: Extending zato-apitest

    Given address "http://apitest-demo.zato.io"
    Given URL path "/demo/json"
    Given format "JSON"
    Given request is "{}"
    Given path "/day" in request is a weekday after "Fri"

    When the URL is invoked

    Then status is "200"

Naming conventions

The name ‘Zato’, case insensitive, cannot be used anywhere in your tests. Don’t use it as a prefix, suffix or anywhere else. This applies to step names, variables, functions, anything. This is a system name reserved for the tool’s own purposes.

Available steps and assertions

Keep in mind that the very first assertion starts with “Then” but then all the following one start with “And” instead of “Then”.

Pattern Notes
Given address “{address}” An address of the API to invoke
Given Basic Auth “{username}” “{password}” Username and password for HTTP Basic Authentication
Given URL path “{url_path}” URL path to invoke
Given HTTP method “{method}” HTTP method to use for invoking
Given request file “{name}” is “{value}” Specifies the path to a file upload, relative to the ./features/form/request directory. This will send the request with http header Content-Type: multipart/form-data; boundary=<boundary>
Given request param “{name}” is “{value}” This will send the request with HTTP header Content-Type: application/x-www-form-urlencoded
Given format “{format}” Either ‘JSON’ or ‘POST’
Given user agent is “{value}” User-Agent string to use
Given header “{header}” “{value}” Arbitrary HTTP header to provide to the API
Given request “{request_path}” Name of a file the request is kept in. ./features/json/request or ./features/form/request will be prepended automatically.
Given request is “{data}” Request to use, inlined.
Given query string “{query_string}” Query string parameters in format of ?a=1&amp;b=2, including the question mark
Given date format “{name}” “{format}” Stores a date format format under a label name for use in later assertions
Given I store “{value}” under “{name}” Stores an arbitrary value under a name for use in later assertions
Given path “{path}” in request is “{value}” Sets path to a string value in the request
Given path “{path}” in request is an integer “{value}” Sets path to an integer value in the request
Given path “{path}” in request is a float “{value}” Sets path to a float value in the request
Given path “{path}” in request is a list “{value}” Sets path to a list value in the request
Given path “{path}” in request is a random string Sets path to a randomly generated string in the request
Given path “{path}” in request is a random integer Sets path to a randomly generated integer in the request
Given path “{path}” in request is a random float Sets path to a randomly generated float in the request
Given path “{path}” in request is one of “{value}” Sets path to a randomly chosen string out of value in the request
Given path “{path}” in request is a random date “{format}” Sets path to a randomly generated date using format format
Given path “{path}” in request is a random date after “{date_start}” “{format}”  
Given path “{path}” in request is a random date before “{date_end}” “{format}”  
Given path “{path}” in request is now “{format}” Sets path to now in local timezone, using format format
Given path “{path}” in request is UTC now “{format}” Sets path to now in UTC, using format format
Given path “{path}” in request is a random date between Sets path to a randomly generated date between date_start and date_end, using format format
“{date_start}” and “{date_end}” “{format}”  
When the URL is invoked Invokes the HTTP-based API under test
Then status is “{status}” Asserts that the HTTP status code in response is status
Then context is cleaned up Cleans up any scenario-specific data, such as namespaces or date formats. Without it, they are carried over to subsequent scenarios
Then header “{header}” is “{value}” Asserts that a header exists and has value value
Then header “{header}” is not “{value}” Asserts that a header exists and does not have value value
Then header “{header}” contains “{value}” Asserts that a header exists and contains substring value
Then header “{header}” does not contain {value}” Asserts that a header exists and does not contain substring value
Then header “{header}” exists Asserts that a header exists, regardless of its value
Then header “{header}” does not exist Asserts that a header does not exist
Then header “{header}” is empty Asserts that a header exists and is an empty string
Then header “{header}” is not empty Asserts that a header exists and is any non-empty string
Then header “{header}” starts with “{value}” Asserts that a header exists and starts with substring value
Then header “{header}” does not start with {value}” Asserts that a header exists and does not start with substring value
Then header “{header}” ends with “{value}” Asserts that a header exists and ends with substring value
Then header “{header}” does not end with “{value}” Asserts that a header exists and does not end with substring value
Then I store “{path}” from response under “{name}” Stores values of paths under labels for the duration of a single test case (feature)
Then I store “{path}” from response under “{name}”, default “{default}” Like above but default value will be used if the response does not contain the provided path
Then path “{path}” contains “{value}” Asserts that value is in the path, that it is its substring
Then path “{path}” is “{value}” Asserts that path is of the value provided
Then path “{path}” is a float “{value}” As above, confirming additionally that value is a float
Then path “{path}” is a list “{value}” As above, confirming additionally that value is a list
Then path “{path}” is an integer “{value}” As above, confirming additionally that value is an integer
Then path “{path}” is empty Asserts that path exists and that it is an empty string
Then path “{path}” is one of “{value}” Asserts that path contains a value from the value elements
Then path “{path}” is not one of “{value}” The opposite of the above
Then path “{path}” is not empty Asserts that path exists with any value
Then response is equal to “{expected}” Asserts that the entire response is equal to expected
Then response is equal to that from “{path}” Asserts that the entire response is equal to what is found in the file pointed to by path