Hi @cfccai,
To call and run a Fabric pipeline from a Notebook while ensuring all actions happen within a single Spark session, you can use the following approach:
Steps
1. After you ensuring that the Change Data Feed (CDF) setting is applied at the beginning of the session:
spark.conf.set('spark.databricks.delta.properties.defaults.enableChangeDataFeed', 'true')
2. Use Azure Data Factory (ADF) REST API: Trigger pipelines using the ADF REST API. You can use requests in Python or other libraries to call the pipeline.
Python Code to Trigger Pipeline:
import requests
# Azure Active Directory token
def get_access_token(tenant_id, client_id, client_secret):
url = f"https://login.microsoftonline.com/{tenant_id}/oauth2/token"
payload = {
'grant_type': 'client_credentials',
'client_id': client_id,
'client_secret': client_secret,
'resource': 'https://management.azure.com/'
}
response = requests.post(url, data=payload)
response.raise_for_status()
return response.json().get('access_token')
# Trigger Fabric pipeline
def trigger_pipeline(subscription_id, resource_group, factory_name, pipeline_name, token, parameters={}):
url = f"https://management.azure.com/subscriptions/{subscription_id}/resourceGroups/{resource_group}/provide..."
headers = {
'Authorization': f'Bearer {token}',
'Content-Type': 'application/json'
}
response = requests.post(url, headers=headers, json=parameters)
response.raise_for_status()
return response.json()
# Fill in your Azure credentials
tenant_id = "your-tenant-id"
client_id = "your-client-id"
client_secret = "your-client-secret"
subscription_id = "your-subscription-id"
resource_group = "your-resource-group"
factory_name = "your-factory-name"
pipeline_name = "your-pipeline-name"
# Get the access token
token = get_access_token(tenant_id, client_id, client_secret)
# Trigger the pipeline
run_response = trigger_pipeline(subscription_id, resource_group, factory_name, pipeline_name, token)
print(f"Pipeline run ID: {run_response.get('runId')}")
Monitor the Pipeline (Optional): If needed, you can monitor the pipeline execution by polling the status using the ADF REST API.
Note:
Ensure Tables Are Created in Current Session: Since the pipeline is external, you may need to verify the tables are properly created within the same Spark session by running checks or integrating additional validation logic in the Notebook.
Ensure proper access and authentication for the REST API call.
If you found this post helpful, please consider marking it as "Accept as Solution" and select "Yes" if it was helpful. help other members find it more easily.
Thank you,
Pavan.
