Python Microsoft Fabric - Picking up results when a notebook finishes

Run a notebook and read what it wrote, or let Fabric call your service the moment it is done.

A notebook - a page of Spark code that runs in Fabric - often ends by writing its results to a table, and a system outside Fabric needs those rows. This page gets the rows from the table to a service that passes them on.

There are two ways to know that the notebook is done. Either Zato starts the notebook and checks on it until it completes, or Fabric runs the notebook on its own schedule and calls Zato when it is finished. Both are below, and the table they read is the same.

The table the notebook writes

Whichever way it is run, the notebook writes the table reminder_candidates:

Reading it is one query, and both ways below end with the same service doing it:

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class ReadResults(Service):

    name = 'results.read'

    def handle(self):

        workspace_id = '11111111-2222-3333-4444-555555555555'
        lakehouse_id = '66666666-7777-8888-9999-000000000000'

        conn = self.microsoft.fabric['My Fabric']

        sql = 'select appointment_id, location, starts_at from reminder_candidates'
        rows = conn.query(workspace_id, lakehouse_id, sql)

        for row in rows:
            appointment_id = row['appointment_id']
            location = row['location']
            starts_at = row['starts_at']
            self.logger.info(f'Reminder for {appointment_id} at {location}, {starts_at}')

        self.response.payload = {'rows': len(rows)}

Zato runs the notebook

The first service starts the notebook and keeps the job's ID in the cache. A second service, scheduled every two minutes, looks the job up, and once it has completed, invokes results.read and forgets the job.

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

class StartNotebook(Service):

    name = 'results.start-notebook'

    def handle(self):

        workspace_id = '11111111-2222-3333-4444-555555555555'
        notebook_id = 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee'

        conn = self.microsoft.fabric['My Fabric']

        # Start the notebook - the call returns as soon as Fabric has accepted the job ..
        job_id = conn.run_job(workspace_id, notebook_id, 'RunNotebook')

        # .. and remember which job to look for.
        self.cache.set('results.notebook_job', job_id)

        self.response.payload = {'job_id': job_id}

# -*- coding: utf-8 -*-

# Zato
from zato.server.service import Service

_running = 'NotStarted', 'InProgress'

class CheckNotebook(Service):

    name = 'results.check-notebook'

    def handle(self):

        job_id = self.cache.get('results.notebook_job')

        # Nothing is running, nothing to check.
        if not job_id:
            return

        workspace_id = '11111111-2222-3333-4444-555555555555'
        notebook_id = 'aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee'

        conn = self.microsoft.fabric['My Fabric']

        job = conn.get_job(workspace_id, notebook_id, job_id)
        status = job['status']

        # Still going - look again in two minutes.
        if status in _running:
            return

        # Whatever it ended with, this job is not looked at again.
        self.cache.delete('results.notebook_job')

        # The notebook completed and the table is ready ..
        if status == 'Completed':
            self.invoke('results.read')

        # .. or it did not.
        else:
            self.logger.warning(f'Notebook job {job_id} ended with {status}')

Schedule results.start-notebook once a day at the time the results are needed and results.check-notebook every two minutes, both under Scheduler → Config in the Dashboard. While the notebook runs, Fabric's monitoring hub shows the job going from In progress to Completed:

Fabric calls Zato

When the data team already runs the notebook from a pipeline on Fabric's own schedule, nothing needs to start it from outside. Instead, the pipeline gets one more step after the notebook - a Web activity that posts to a REST channel in Zato, and the channel's service is results.read.

Under Connections → REST → Channels, create the channel with the URL path /api/fabric/results-ready, the service results.read and an API key. In enmasse YAML:

security:
  - name: Fabric Pipeline Key
    type: apikey
    username: fabric-pipeline
    password: Zato_Enmasse_Env.FabricPipelineKey

channel_rest:
  - name: api.fabric.results-ready
    service: results.read
    url_path: /api/fabric/results-ready
    security: Fabric Pipeline Key
    data_format: json

In the pipeline, add a Web activity after the notebook activity, connected on success, with the channel's full URL, method POST, the header X-API-Key set to the key, and an empty JSON object as the body:

From then on, the moment the notebook completes, results.read runs.

Page	What it covers
Running a pipeline and refreshing a report	Running a job and waiting for it inside one service
Letting Fabric pipelines read your local systems	The other thing a pipeline can do with a REST channel
Jobs	run_job, get_job and cancel_job in detail

Learn more

Development documentation Everything about writing services, in one place

Requests and responses What a service receives, what it returns and how to shape both

Integration examples Ready-made code for the systems you are likely to connect to

IDE and debugging Write services in the Dashboard or in your own editor

Data models Declare inputs and outputs and have them validated for you

In-depth API tutorial The full platform tutorial, from installation to production patterns