Version 3.3 Documentation
Blog · REST · Security · SSL-TLS · Subscribe Click to subscribe to insights
Subscribe Click to subscribe to insights

Resilient REST APIs with SSL/TLS client certificates

2021-05-10, by Dariusz Suchojad

This Zato how-to is about ensuring that only API clients with valid SSL/TLS certificates, including expected certificate fingerprints or other metadata, can invoke selected REST endpoints. In this way, we are making access to the endpoints secure and, at the same time, we can guard against a class of faults related to the Certificate Authority infrastructure.

Overview

Given that in front of all Zato servers in a cluster is a load-balancer, it is the load-balancer that accepts API client connections.

The load-balancer needs to be configured to require clients to present their certificates. Moreover, we will configure the load-balancer to extract parts of the certificates (e.g. fingerprints) that will be validated by Zato servers to ensure that the Certificate Authority which issues certificates has not issued an unknown certificate.

Prerequisites

What we need for the setup above is:

  • An SSL/TLS private key for the load-balancer (lb.key.pem)
  • A certificate associated with the load-balancer's private key (lb.cert.pem)
  • An SSL/TLS private key for an API client (client.key.pem)
  • A certificate associated with the client's private key (client.cert.pem)
  • A certificate of the Certificate Authority that issued the load-balancer's certificate (lb.ca.pem)
  • A certificate of the Certificate Authority that issued the clients's certificate (client.ca.pem)

Both of the certificates can be signed by the same Certificate Authority (CA) or the CAs may be different, there is no difference, and whether it is either case will depend on particular circumstances.

For instance, if you are integrating applications within a single company - or, in general, if all the components of the infrastructure are under your control - then there will likely already exist an internal CA issuing certificates for addresses in private ranges (e.g. 10.x.x.x) and the same CA will issue both certificates.

On the other hand, if the API clients are applications external to your infrastructure, you may be in a situation where the CA which issued the client certificate is not under your control, in which case you may be required to consider to what degree it can be trusted. We will cover it later on.

Load-balancer's configuration

Make sure that you have lb.key and lb.cert files and save them in a directory of choice in the same systems that the load-balancer runs in, e.g. let's say that you save it to /mypath and you have two full paths:

  • /mypath/lb.key.pem
  • /mypath/lb.cert.pem

The files need to be concatenated now and the result will be one, combined file with both the load-balancer's certificate and its private key:

$ cat /mypath/lb.cert.pem > /mypath/lb.pem
$ cat /mypath/lb.key.pem  >> /mypath/lb.pem

Ensure that you have the certificate of the CA that signed the certificate of the API clients that will be connecting to Zato - let's assume that you save it to /mypath/client.ca.pem.

In Zato web-admin, go to Clusters, then to Load-balancer and next to Config source code view.

At the end of the file, enter a new section as below, then click "Validate and save".

frontend front_tls_fields

  mode http
  default_backend bck_http_plain
  option forwardfor
  reqadd X-Forwarded-Proto:\ https

  acl has_x_forwarded_proto req.fhdr(X-Forwarded-Proto) -m found
  http-request deny if has_x_forwarded_proto

  bind 0.0.0.0:51223 ssl crt /mypath/lb.pem verify required ca-file /mypath/client.ca.pem

  http-request set-header X-Zato-TLS-Fingerprint %{+Q}[ssl_c_sha1,hex]

The load-balancer that Zato uses is an embedded HAProxy instance and the configuration above has several interesting parts

  • 0.0.0.0:51223 - we will accept encrypted client connections on port 51223 from any interface

  • crt /mypath/lb.pem - points to the file with the concatenated contents of both the load-balancer's certificate and its private key

  • verify required - API callers are required to have client certificates

  • ca-file /mypath/client.ca.pem - path to the certificate of the CA that signed the certificate of the API client; if the client certificate was signed by a different CA, a request from such a client will be rejected

  • http-request set-header - this line extracts the client's certificate fingerprint and creates a new HTTP header, X-Zato-TLS-Fingerprint, that is sent to Zato servers. More about it in a few steps.

Testing the initial configuration

We can already test the load-balancer's configuration from command line, using curl. Note that if the CA that signed the load-balancer's certificate is not among ones that curl recognizes, the "-k" option is needed, as below.

First, let's check what happens if we do not use a client certificate:

$ curl -k https://localhost:51223/zato/ping
curl: (56) OpenSSL SSL_read: routines:ssl3_read_bytes:tlsv13 alert certificate required
$

Good, the request was rejected because the endpoint invoked (the load-balancer) required a certificate but we did not send any.

Now, let's invoke it again, this time around with a client certificate along with its private key.

$ curl -k https://localhost:51223/zato/ping \
  --cert ./client.cert.pem \
  --key  ./client.key.pem \
  ; echo
{"pong":"zato","zato_env":{"result":"ZATO_OK","cid":"c37553014674dbb3884efbf0","details":""}}
$

The need for more

In most integration scenarios, this would have been all - the load-balancer is configured to require client certificates, clients send certificates as requested, and Certificate Authorities issue their certificates to all the parties that need to integrate.

However, what if a CA issues an invalid certificate and it is not aware of it or if the certificate is valid but it is not used as expected?

This happens quite often in practice - for instance, there may be a single CA in the organization and it issues certificates for all kinds of environments, such as development, testing and production. If the load-balancer merely accepts all certificates issued by this CA, it is possible that development environments will be allowed to connect to production.

Another common example are client certificates issued by third-party CAs about which you know nothing, that are just pieces of an external infrastructure. You have no reason to trust them fully and you have no guarantees as to what certificates they will issue, to whom, why and what for.

In short, there may be a need to be able to do more than only configuring the load-balancer to trust a particular CA or CAs.

The technique employed in such situations is called certificate pinning - you pin a specific certificate to a REST endpoint and if that certificate ever changes, even if the CA that signed it is the same, the request will be rejected.

This is why in the configuration above we extracted the client certificate's fingerprint - the fingerprint is what uniquely identifies a particular certificate.

Hence, in the next steps we are going to create a new security definition, make it point to a specific SSL/TLS client certificate's fingerprint and then have an API service mounted on a REST channel be accessible only if the fingerprint matches.

Extracting metadata from client certificates

We begin by finding the fingerprint of the certificate using OpenSSL - as with other elements of SSL/TLS metadata, it can be extracted from command line:

$ openssl x509 -text \
  -in ./client.cert.pem \
  -fingerprint | grep -E 'Subject:|Fingerprint'

Subject: C = US, ST = My State, O = My Company, OU = client.api, CN = localhost
SHA1 Fingerprint=E3:25:11:E8:CF:F9:92:45:10:58:F4:B7:0F:FB:F9:01:7B:FD:0B:2A
$

Above, we extracted the fingerprint but we also did the same for the certificate's Subject, i.e. the party that the certificate was issued to - this is purely to confirm that we are working with the right certificate.

Adding a security definition

Now, in Zato web-admin, go to Security -> SSL/TLS -> Channels and fill out the form:

Things to observe:

  • This kind of a security definition is built around the idea of conditions

  • Each line is a key=value pair, representing a condition that needs to be satisfied in runtime

  • Keys and values are extracted by the load-balancer to be sent in HTTP headers

  • Zato strips the "X-Zato-TLS-" prefix and uppercases the remaining part

  • This is why X-Zato-TLS-Fingerprint is available as a "FINGERPRINT" key in the definition

  • In this case, we are only checking the fingerprint but we could just as well extract other pieces of metadata from the certificate and give the security definition access to them

  • Finally, the fingerprint uses colons under openssl but they are not used in Zato

Taken together:

  • The definition has a list of definitions

  • When a request with a client certificate arrives, the load-balancer, having already validated that the client certificate is signed by a CA it is configured to recognize, extracts certificate metadata and forwards it to Zato servers in HTTP headers

  • Zato consults the forwarded headers and compares them with conditions

  • If any condition is not satisfied, the request is rejected

  • If all conditions are satisfied, the request is allowed

Naturally, in this setup, you need to know in advance about changes to the client certificate and update your security definition accordingly.

With that knowledge, we can now add a test API service and create a REST channel for it - the channel will be secured with the security definition created above.

Adding a test API service

For illustration purposes, we can deploy a service like the one below which simply returns static responses - it will be added to a REST channel in the next step.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class GetData(Service):
    name = 'api.customer.data'

    def handle(self):

        self.response.payload = {
            'name': 'John Doe',
            'id': '123'
        }

Creating a REST API channel

At this point, we have a security definition and we have service so we only need a REST channel that the service will be mounted on. The channel will have its security definition set to the one that we defined earlier.

In web-admin, navigate to Connections -> Channels -> REST and fill out the form as below, observing in particular what goes to Service and Security definition fields. Note also that the channel's data format is set to JSON.

Testing the final configuration

First, let's start again by checking what happens if the client sends invalid information - in this case, we will make it use an invalid certificate, one that is signed by the CA that the load-balancer trusts but not one that has a valid fingerprint.

$ curl -k https://localhost:51223/api/customer \
  --cert ./invalid-client.cert.pem
  --key  ./invalid-client.key.pem \
  ; echo
{"result":"Error",
 "cid":"ccb4f2111c0a21c1150a42f1",
 "details":"You are not allowed to access this resource"}
$

We get an error and this is a good sign because it means that the certificate was accepted by the load-balancer yet the precondition (valid fingerprint) was not recognized by the server.

The next natural step is to invoke the REST endpoint with an expected client certificate:

$ curl -k https://localhost:51223/api/customer \
  --cert ./client.cert.pem
  --key  ./client.key.pem \
  ; echo
{"name":"John Doe","id":"123"}
$

That works as expected too - we have a business response on output and we can be sure that not only has been a client certificate signed by a known CA used but it is the exact certificate that we expect to receive from the client of this particular REST API.

More blog posts➤