Pass Information to Notebook from Invoke Pipeline

Hi esjetmore,

Currently the new Invoke Pipeline (Preview) activity is having issues with Pipeline return value. Once it is available, you can use it to return values from the invoked Pipelines. 

Invoke pipeline activity - Microsoft Fabric | Microsoft Learn

Before existence of Pipeline return value and Fail Activity in ADF, we used to copy empty files with specific name from one location to another when a particular pipeline finishes. We then check the existence of file to know which pipelines successfully completed before starting another process. You can try that with Lakehouse. 

Or, Pipeline REST API is simple if you can pass all the pipeline Run IDs as an array (using set Variable) to the notebook and in the notebook you can call the Fabric REST API using the below code. 

#Helper function to call Fabric REST API

from notebookutils import mssparkutils as msu
import requests
import time

def call_fabric_api(method, uri, payload=None):
    endpoint = "https://api.fabric.microsoft.com/v1"

    # Get PBI Access token and have it in the header for Authorization
    headers = {
        "Authorization": "Bearer " + msu.credentials.getToken("pbi"),
        "Content-Type": "application/json"
    }

    #Create new session and try sending a request to the Fabric REST API
    session = requests.Session()
    try:
        url = f"{endpoint}/{uri}"
            
        response = session.request(method, url, headers=headers, json=payload, timeout=120)
        print(response.json())
        return response.json()

    except requests.RequestException as ex:
        print(ex)

And using something like this:

method = "get"

base_uri = f"workspaces/<your WS Id>/items/<pipeline id>/jobs/instances/<job ID>"
data = call_fabric_api(method,base_uri)

if data["status"] == "Completed":
# Add your logic here

https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-rest-api#get-item-job-instance