Forum Discussion

b_schmitt's avatar
b_schmitt
Regular Visitor
1 year ago
Solved

Questions about functionality about the Apache Airflow Job in Fabric

Hello,   I want to set up Apache Airflow Jobs in Fabric and have a few questions.   1. Is there a way to trigger a DAG via an HTTP-Trigger, REST-API or something other than timed schedules? The b...
  • v-echaithra's avatar
    1 year ago

    Yes, it is possible to trigger a Fabric Pipeline using Azure Logic Apps:

    1.  Start by creating a new Logic App in the Azure portal. You can choose a trigger that suits your scenario, such as an HTTP request, a schedule, or an event.
    2.  In the Logic App, add an action to call the Fabric Pipeline. This can be done using the HTTP action to make a REST API call to the Fabric Pipeline endpoint.
    3. Ensure your Fabric Pipeline is set up to accept triggers from external sources. You might need to configure authentication and permissions to allow the Logic App to trigger the pipeline.

    Reference: https://learn.microsoft.com/en-us/answers/questions/1805816/accessing-fabric-lakehouse-via-logic-app

     

    When triggering a DAG from the CLI, the REST API or the UI, it is possible to pass configuration for a DAG Run as a JSON blob.

    Check these Airflow settings for parallel runs: 
    Max number of concurrent tasks across the whole Airflow environment,
    Max number of concurrent tasks per DAG,
    if you are you using a Pool to limit concurrency, check if both tasks are in a pool that has only 1 slot, That can cause serial execution.
    What executor are you using?
    If you’re on SequentialExecutor, that’s the bottleneck.

    Thanks and Regards
    Chaithra