FHIR bulk export

A bulk export pulls a whole population out of a FHIR server at once - every patient of a group, every patient the server holds, or everything on the server - instead of reading resources one at a time. The server packs the data into NDJSON files, one resource per line, and Zato downloads them and delivers each file to the destinations you choose. The export is the $export operation of the FHIR Bulk Data Access specification, which Epic, Oracle Health, Medicare's BCDA and the other population health APIs implement.

You configure an export on an outgoing FHIR connection and it runs on a schedule, from the Dashboard whenever you click Run now, or from a service through client.export().

Configure an export

Open the connection under Connections > Outgoing > HL7 > FHIR - or create one as the connections page shows - and switch to the Bulk export tab:

1. Active: on - the schedule below runs only while this is on
2. Export: Group
3. Group ID: the ID of the Group resource on the server, e.g. diabetes-registry
4. Resource types: Patient, Observation, Condition
5. Run every: 1 and days
6. Destinations: pick a type and a connection, then click Add
7. Click OK

The first export starts at the Start time, or right away when it is empty, and every export after it runs Run every later. Each export is one job with an ID of its own that you can follow on the Bulk exports page described below.

Servers differ in which of the three levels and which parameters they implement - Epic, for instance, exports groups only, and BCDA exports the groups it defines for each organisation. The server's documentation says what it supports and a request it does not understand fails the job with the server's answer in the audit log.

Choose where the files go

A destination is a connection, or a service, that receives the exported data. An export can have several of them and each file goes to every one of them in order. The picker on the tab offers four types:

To add a destination, pick its type, pick the connection or the service, fill in the options the type has - SFTP takes the remote path - and click Add. Each destination is a badge on the tab and its x removes it. A destination refusing a file stops the export at that file, and the file is offered again when the export runs next, so a destination that was down for a while receives everything it missed.

Receive the files in a service

A service is a Python class that Zato runs when something invokes it. A service named as a destination is invoked once for each file of the export and self.request.input is the file. Iterating over it yields the resources one at a time, each already parsed, so a file of any size is read without being loaded into memory:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class LoadRegistry(Service):
    """ Receives one file of a bulk export and writes each resource to the registry.
    """
    name = 'demo.fhir.load-registry'

    def handle(self) -> 'None':

        # The file - which export it belongs to and what it holds ..
        file = self.request.input

        self.logger.info('Received %s with %s %s resources', file.file_name, file.count, file.resource_type)

        # .. a file of errors says what the server could not export ..
        if file.is_error_file:
            for outcome in file:
                self.logger.warning('Export issue -> %s', outcome['issue'])
            return

        # .. and every other file is the resources themselves.
        for resource in file:
            self.logger.info('%s/%s', resource['resourceType'], resource['id'])

The file object carries job_id, connection_name, resource_type, count, file_name, path and is_error_file. Besides iterating over it for parsed resources, file.lines() yields the raw lines as bytes, which is what to pass on when the resources only need to be forwarded rather than read.

An export may produce files of OperationOutcome resources alongside the data - these list the resources the server could not export and why. They are delivered like any other file, with is_error_file set, so the service above decides what to do with them.

Run an export on demand

Each connection's row on the FHIR connections page has a Bulk exports link that opens the page of its exports. The page lists the recent jobs with when each started and finished, its status, how many files and resources it delivered and how many errors it met, and Run now starts an export with the tab's settings right away, schedule or no schedule.

A service starts an export through the connection's client, which returns the ID of the job it started. What is given to the call replaces the tab's settings for that one export and whatever is left out comes from the tab - the destinations included, unless the call names destinations of its own:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class ExportRegistry(Service):
    """ Starts a bulk export of a patient group, with only what changed since the start of the year.
    """
    name = 'demo.fhir.export-registry'

    def handle(self) -> 'None':

        client = self.fhir['FHIR.Epic']

        job_id = client.export(
            group_id='diabetes-registry',
            types=['Patient', 'Observation', 'Condition'],
            since='2026-01-01T00:00:00Z',
        )

        self.logger.info('Started export %s', job_id)

The call takes level - group, patient or system - with group_id or patient_ids as the level needs, and types, since, type_filter and destinations, each in the shape the tab's field of the same name takes. The call returns as soon as the export is started, with the files arriving at the destinations as the server produces them.

Create the group

A group export needs a Group resource on the server listing the patients to export. Some servers maintain these groups themselves - Epic exports the groups built in its own tools, BCDA defines one per organisation - while others let a client create them. Where the server allows it, a Group is a resource like any other and the connection creates it with the same call that creates a Patient:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class CreateRegistry(Service):
    """ Creates the Group resource a bulk export reads its patients from.
    """
    name = 'demo.fhir.create-registry'

    def handle(self) -> 'None':

        client = self.fhir['FHIR.Sample']

        group = client.resource('Group')
        group.id = 'diabetes-registry'
        group.type = 'person'
        group.actual = True
        group.member = [
            {'entity': {'reference': 'Patient/1001'}},
            {'entity': {'reference': 'Patient/1002'}},
        ]

        group.save()

The resources page covers everything about creating and updating resources through the connection.

Define exports in YAML

Instead of filling out the tab, you can declare the export in YAML and import it with enmasse, under the connection's bulk_export key. The lists are YAML lists and each destination names its type, its connection and the options of the type:

outgoing_fhir:

  - name: FHIR.Epic
    address: https://fhir.epic.com/interconnect-fhir-oauth/api/FHIR/R4
    security: FHIR.Identity
    bulk_export:
      is_active: true
      level: group
      group_id: diabetes-registry
      types:
        - Patient
        - Observation
        - Condition
      since: "2026-01-01T00:00:00Z"
      run_every: 1
      run_unit: days
      destinations:
        - name: archive
          type: sftp
          connection: SFTP.Archive
          options:
            remote_path: /exports/{job_id}/{file_name}
        - name: registry
          type: service
          connection: demo.fhir.load-registry

Every key is listed in the enmasse reference.

Watch an export

The Bulk exports page shows each job of a connection with its status - Running with the step it is at, Done or Failed - and how many files, resources and errors it has so far. Each row's Audit log link opens the audit log filtered to that job, under the fhir-bulk-export source, where every step is one entry - the request that started the export, the server's manifest, each file downloaded with how many resources it held, each delivery to each destination and, at the end, the export's total.

A failed job stays listed with what went wrong in its row and in the audit log. Starting the export again, from Run now or on its schedule, runs a new job.

Troubleshooting

The job fails right after it starts. The server refused the $export request and its answer is in the audit log entry of the job's first step. The usual causes are a level the server does not implement - Epic, for one, takes group exports only - a Group ID that does not exist, or OAuth scopes that do not cover bulk data, which with SMART Backend Services are the system/*.read scopes of the types exported.

The job runs for a long time. The server builds the export at its own pace and says how long to wait between polls - large populations take minutes to hours on production EHRs. The job's row shows the step it is at and the server's progress message when the server sends one.

A destination refuses a file. The export stops at that file and its row shows the destination's answer. Once the destination is reachable again, running the export once more delivers the file and the ones after it.

The data is incomplete. An OperationOutcome file lists what the server could not export - deliver the error files to a service to read them, or look at the job's audit log entries, where each such file is recorded with its count.

Only new data is wanted. Set Since to the moment of the previous export, or pass since to client.export() from a service that remembers when it last ran. Servers that implement it return only the resources changed after that moment.

See also

Learn more