Python Microsoft Fabric - Data science
ML models and experiments in Fabric, with training triggered through notebook jobs.
Fabric's data science workload tracks machine learning work as two item types - experiments record training runs, and ML models version the resulting models, both backed by MLflow. Your services manage them like any other items and trigger the training itself through notebooks. You create a Fabric connection in the Dashboard and all of it is available to your services.
Listing ML models
ML models are workspace items of the MLModel type.
# -*- coding: utf-8 -*-
# Zato
from zato.server.service import Service
class ListMLModels(Service):
input = 'workspace_id'
def handle(self):
# Get the connection by its Dashboard name
conn = self.microsoft.fabric['My Fabric']
# List only the ML model items
response = conn.list_items(self.request.input.workspace_id, 'MLModel')
models = []
for item in response['value']:
models.append({
'id': item['id'],
'name': item['displayName'],
})
self.response.payload = {'models': models}
Listing experiments
Experiments are items of the MLExperiment type - each one groups the training runs of a single piece of ML work.
# -*- coding: utf-8 -*-
# Zato
from zato.server.service import Service
class ListExperiments(Service):
input = 'workspace_id'
def handle(self):
conn = self.microsoft.fabric['My Fabric']
response = conn.list_items(self.request.input.workspace_id, 'MLExperiment')
experiments = [item['displayName'] for item in response['value']]
self.response.payload = {'experiments': experiments}
Creating an experiment
New ML work starts with an experiment - created like any other item.
# -*- coding: utf-8 -*-
# Zato
from zato.server.service import Service
class CreateChurnExperiment(Service):
input = 'workspace_id'
def handle(self):
conn = self.microsoft.fabric['My Fabric']
experiment = conn.create_item(
self.request.input.workspace_id,
'Churn prediction',
'MLExperiment',
description='Customer churn model training runs',
)
self.response.payload = {'experiment_id': experiment['id']}
Triggering model training
Training runs inside a notebook - the notebook trains the model, logs the run to the experiment with MLflow, and registers the resulting model. Your service starts it as a notebook job, passing the training parameters.
# -*- coding: utf-8 -*-
# Zato
from zato.server.service import Service
class RetrainChurnModel(Service):
input = 'workspace_id', 'training_notebook_id'
def handle(self):
conn = self.microsoft.fabric['My Fabric']
# Parameters the training notebook will receive
payload = {
'executionData': {
'parameters': {
'experiment_name': {'value': 'Churn prediction', 'type': 'string'},
'model_name': {'value': 'churn-model', 'type': 'string'},
}
}
}
# Start the training notebook
conn.run_job(
self.request.input.workspace_id,
self.request.input.training_notebook_id,
'RunNotebook',
payload,
)
self.response.payload = {'status': 'training started'}
A common trigger for this service is a pipeline completing a data load, or a schedule - retraining every night on fresh data.
Reading model metadata
conn.get_item returns an ML model's metadata - its name, description and identifiers - which is how services report on what models exist and when they last changed.
# -*- coding: utf-8 -*-
# Zato
from zato.server.service import Service
class GetModelDetails(Service):
input = 'workspace_id', 'model_id'
def handle(self):
conn = self.microsoft.fabric['My Fabric']
model = conn.get_item(self.request.input.workspace_id, self.request.input.model_id)
self.response.payload = {
'name': model['displayName'],
'description': model['description'],
}