Managing and Exporting High-Volume Datasets (~1M Records) in Power BI

Hi Gabry,
Thanks for give an idea about how the flow will look like, so I was trying to create a flow had few queries with respect to it:

import json

# 1. Get JSON string from Power Automate (filters parameter)
filters_str = dbutils.widgets.get("filters")
filters = json.loads(filters_str)

print("Received filters:", filters)

# 2. Load data 
df = spark.read.sql("""
    SELECT OrderID, Name, Profit, Quantity, OrderDate, Region
    FROM SalesTable
""")

# 3. Apply filters dynamically
for col, val in filters.items():
    if isinstance(val, list):
        # Multiple selections from slicer
        df = df.filter(df[col].isin(val))
    else:
        # Range handling for dates or numbers
        if "Start" in col:
            base_col = col.replace("Start", "")
            df = df.filter(df[base_col] >= val)
        elif "End" in col:
            base_col = col.replace("End", "")
            df = df.filter(df[base_col] <= val)
        else:
            # Single value equality
            df = df.filter(df[col] == val)

# 4. Save to OneLake / Blob
output_path = "abfss://[email protected]/SalesExports/FilteredExport.csv"

(df
 .coalesce(1)  # single file
 .write
 .mode("overwrite")
 .option("header", "true")
 .csv(output_path))

print("✅ Export completed:", output_path)

The above code was for the Fabric notebook (to accept dynamic range of filters which means there can be n slicers that can be added later so the code should not fail if newer one's are added). Ovver here I was not sure what should be added as the output_path (should I add the URL of lakehouse)

Then coming to the Power Automate flow:

Then in the compose action I was using the JSON body to build the Request body which looks like:

{
  "notebookExecution": {
    "parameters": {
      "filters": "@{json(triggerBody()?['filters'])}"
    }
  }
}

Then to run the Fabric Notebook using the HTTP action (premium connector), I did across two URL that can run a notebook:
1st: https://api.fabric.microsoft.com/v1/workspaces/{{WORKSPACE_ID}}/items/{{ARTIFACT_ID}}/jobs/instances (Received from ChatGPT)
2nd: (which is been used currently and here we don't need to pass anything in the body i.e. empty)
https://api.fabric.microsoft.com/v1/workspaces/{workspace_id}/items/{artifact_id}/jobs/instances?jobType=RunNotebook
But the issue is I am not sure how to pass the filters (Compose action output if I used the 1st URL and to generate the token do I need to register an app (how to get the token))
-> The next step was to poll the notebook untill it returns 200 (succeded) using the GET request within it and which uses the status code from the previous HTTP action, but here I needed o get the JobInstanceID not sure on how to get that

# GET URL:
GET https://api.fabric.microsoft.com/v1/workspaces/{{WORKSPACE_ID}}/items/{{ARTIFACT_ID}}/jobs/instances/{{jobInstanceId}}

Right now I have the workspace and ARtifact id which we can find in the URL of the Fabric notebook.

ANd the next steps were to the notebook should save the filtered data in OneLake and using GET link to retrieve the link of the file stored (location) and finally using send email action notify user.

So can you help me out here and is the above flow design correct?.

Poojara_D12 , v-hashadapu : If you have anything to to do please do share them as well.
Regards,
Sidhant