# Python Microsoft Fabric - Running a pipeline and refreshing a report

After the nightly load, run the pipeline, wait for it, refresh the report's data and tell the team.

Once the nightly load has put the day's rows into the lakehouse, two more things have to happen before a report is current. A pipeline - a sequence of steps built in Fabric - reshapes the raw rows into the tables the report reads, and then the report's data has to be refreshed, because a Power BI report shows what it loaded last, not what the tables hold now. This page chains both in one service that runs right after the [nightly load](https://zato.io/docs/dev/examples/cloud/fabric/loading-tables.html).

> **What you need from your Fabric admin**
>
> The workspace ID, from the address bar. The pipeline's item ID, from the address bar when the pipeline is open, after `/pipelines/`. The ID of the report's dataset, which is the item the report reads its data from - in the workspace list, it is the item with the same name as the report but a different icon, and its ID is in the address bar after `/datasets/` when it is open.

## The pipeline before the run {#the-pipeline-before-the-run}

The pipeline's run history shows last night's run and nothing yet for tonight:

## The service {#the-service}

A pipeline run and a report refresh are both jobs in Fabric, started with `run_job` and checked with `get_job`, so one method, `wait_for_job`, waits for either. It cancels a job that has not ended within a set time.

```python
# -*- coding: utf-8 -*-

# stdlib
from time import sleep

# Zato
from zato.common.typing_ import any_
from zato.server.service import Service

_running = 'NotStarted', 'InProgress'
_check_interval = 20
_pipeline_timeout = 1800
_refresh_timeout = 600
_timed_out = 'TimedOut'

class NightlyPipelineAndReport(Service):

    name = 'nightly.pipeline-and-report'

    def wait_for_job(self, conn:'any_', workspace_id:'str', item_id:'str', job_id:'str', timeout:'int') -> 'str':
        """ Checks a job's status until it ends or the timeout passes, then returns its final status.
        """
        waited = 0

        while waited < timeout:

            job = conn.get_job(workspace_id, item_id, job_id)
            status = job['status']

            # The job has ended ..
            if status not in _running:
                out = status
                break

            sleep(_check_interval)
            waited += _check_interval

        # .. out of time, stop the job.
        else:
            conn.cancel_job(workspace_id, item_id, job_id)
            out = _timed_out

        return out

    def handle(self):

        workspace_id = '11111111-2222-3333-4444-555555555555'
        pipeline_id = 'bbbbbbbb-cccc-dddd-eeee-ffffffffffff'
        dataset_id = 'cccccccc-dddd-eeee-ffff-000000000000'

        conn = self.microsoft.fabric['My Fabric']

        # Run the pipeline and wait for it, up to 30 minutes ..
        pipeline_job_id = conn.run_job(workspace_id, pipeline_id, 'Pipeline')
        pipeline_status = self.wait_for_job(conn, workspace_id, pipeline_id, pipeline_job_id, _pipeline_timeout)

        if pipeline_status != 'Completed':
            text = f'Nightly pipeline ended with {pipeline_status}'
            self.microsoft.teams.send('Ops Teams', to='Data team/Nightly', text=text)
            self.response.payload = {'pipeline': pipeline_status}
            return

        # .. then refresh the report's data and wait for that too, up to 10 minutes ..
        refresh_job_id = conn.run_job(workspace_id, dataset_id, 'DefaultSemanticModelRefresh')
        refresh_status = self.wait_for_job(conn, workspace_id, dataset_id, refresh_job_id, _refresh_timeout)

        # .. and send both statuses to Teams.
        text = f'Nightly pipeline Completed, report refresh {refresh_status}'
        self.microsoft.teams.send('Ops Teams', to='Data team/Nightly', text=text)

        self.response.payload = {'pipeline': pipeline_status, 'refresh': refresh_status}
```

The message to the team goes through a [Microsoft Teams connection](https://zato.io/docs/dev/examples/microsoft-teams.html) created in the Dashboard beforehand.

Schedule the service under `Scheduler → Config` with the cron-style type and `30 2 * * *`, half an hour after the nightly load, so the load has finished before the pipeline starts.

## What Fabric shows afterwards {#what-fabric-shows-afterwards}

The pipeline's run history has tonight's run, submitted by the app registration:

And the dataset's refresh history shows the refresh that followed:

The report now shows what the tables hold after tonight's load.

## See also {#see-also}

- [Loading data into tables](https://zato.io/docs/dev/examples/cloud/fabric/loading-tables.html) - The nightly load this service runs after
- [Picking up results when a notebook finishes](https://zato.io/docs/dev/examples/cloud/fabric/notebook-results.html) - Checking a job's status from a scheduled service
- [Jobs](https://zato.io/docs/dev/examples/cloud/fabric/api/jobs.html) - run\_job, get\_job and cancel\_job, job types and statuses

## Learn more {#learn-more}

- [Development documentation](https://zato.io/docs/dev/) - Everything about writing services, in one place
- [Requests and responses](https://zato.io/docs/dev/request-response/) - What a service receives, what it returns and how to shape both
- [Integration examples](https://zato.io/docs/dev/examples/) - Ready-made code for the systems you are likely to connect to
- [IDE and debugging](https://zato.io/docs/dev/ide/) - Write services in the Dashboard or in your own editor
- [Data models](https://zato.io/docs/dev/model/) - Declare inputs and outputs and have them validated for you
- [In-depth API tutorial](https://zato.io/tutorials/main/01.html) - The full platform tutorial, from installation to production patterns