Airflow Jobs

Hi ArielSubia ,

You're hitting the reality of Fabric Airflow - it's more limited than your local setup, but there are good workarounds for your API-to-dashboard workflow.

For your folder structure issue: Since Fabric only accepts individual .py files, you need to flatten your project:

• Put all your reusable functions in a single utilities.py file
• Import those functions in your DAG files
• Use Fabric's environment variables for configuration instead of config files

For API data storage: Your JSON files can go directly into OneLake (Fabric's data lake):

# In your Airflow DAG
from fabric.lakehouse import LakehouseClient

def store_api_data():
    # Get your API data
    json_data = call_your_api()
    
    # Write to lakehouse
    lakehouse = LakehouseClient("your-workspace", "your-lakehouse")
    lakehouse.upload_file(json_data, f"api-data/file_{datetime.now()}.json")

Better approach - skip files entirely: Instead of storing JSON files, write directly to Delta tables in your lakehouse:

import pandas as pd

def api_to_delta():
    api_response = call_your_api()
    df = pd.json_normalize(api_response)
    # Write directly to Delta table
    df.to_delta("abfss://workspace/lakehouse/Tables/api_data")

For the notebooks/pipelines part: Once data is in your lakehouse, trigger Fabric notebooks from Airflow using the Fabric REST APIs or set up event-driven triggers when new data lands.

The key is treating Fabric as a unified platform rather than trying to replicate your local folder structure.

What kind of API are you working with? That might affect the best storage approach.

If my response resolved your query, kindly mark it as the Accepted Solution to assist others. Additionally, I would be grateful for a 'Kudos' if you found my response helpful.
This response was assisted by AI for translation and formatting purposes.