Questions about functionality about the Apache Airflow Job in Fabric

[b_schmitt]

Hello,

I want to set up Apache Airflow Jobs in Fabric and have a few questions.

1. Is there a way to trigger a DAG via an HTTP-Trigger, REST-API or something other than timed schedules? The best thing would be event-based triggers based on certain data-files.

2. How do I pass parameters to a DAG in Fabric? I don't want to create a single DAG for every entity but instead pass the item_id to the DAG so that it executes it.

3. How do I pass the output of one task as the input of the following task? In regular Airflow I would use XCOMs but I haven't found anything in the Fabric documentation.

4. How do I run several tasks in parallel? I have my flow currently like this:

  [read_2018_data, read_2019_data] >> join_data >> calculate_columns
  read_2019_data >> read_data_failure
  join_data >> join_data_failure
  calculate_columns >> calculate_columns_failure

Are read_2018_data and read_2019_data being run in parallel? It doesn't look like it in the graph overview.

5. Are there good use-case examples for DAGs that can be used in Fabric? I find it to be rather difficult to develop them since I can't find any examples or documentation apart from really basic things that don't really reflect production use-cases.

Any help on this would be greatly appreciated,

Benedikt

[v-echaithra]

Hi @b_schmitt ,

We wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?
If our response addressed, please mark it as Accept as solution and click Yes if you found it helpful.

Regards,
Chaithra.

[v-echaithra]

Hi @b_schmitt,

As i haven't heard back from you, please let me know if the provided answer was helpful, please mark it as "Accept as solution" and click "Yes" if you found it helpful, it will be helpful for other members of the community who have similar problems as yours to solve it faster.

Regards,
Chaithra.

[v-echaithra]

Hi @b_schmitt ,

If the provided answer was helpful, please mark it as "Accept as solution" and click "Yes" if you found it helpful, it will be helpful for other members of the community who have similar problems as yours to solve it faster.

Regards,
Chaithra.

[v-echaithra · Accepted solution]

Yes, it is possible to trigger a Fabric Pipeline using Azure Logic Apps:

1.  Start by creating a new Logic App in the Azure portal. You can choose a trigger that suits your scenario, such as an HTTP request, a schedule, or an event.
2.  In the Logic App, add an action to call the Fabric Pipeline. This can be done using the HTTP action to make a REST API call to the Fabric Pipeline endpoint.
3. Ensure your Fabric Pipeline is set up to accept triggers from external sources. You might need to configure authentication and permissions to allow the Logic App to trigger the pipeline.

Reference: https://learn.microsoft.com/en-us/answers/questions/1805816/accessing-fabric-lakehouse-via-logic-app

When triggering a DAG from the CLI, the REST API or the UI, it is possible to pass configuration for a DAG Run as a JSON blob.

Check these Airflow settings for parallel runs: 
Max number of concurrent tasks across the whole Airflow environment,
Max number of concurrent tasks per DAG,
if you are you using a Pool to limit concurrency, check if both tasks are in a pool that has only 1 slot, That can cause serial execution.
What executor are you using?
If you’re on SequentialExecutor, that’s the bottleneck.

Thanks and Regards
Chaithra

[b_schmitt]

So I need to create an Azure Logic App to trigger DAGs in Fabric via REST API? That seems a bit convoluted

[v-echaithra]

Hi @b_schmitt ,

Yes, it’s a bit indirect but this is the supported approach. Currently Microsoft Fabric does not support triggering DAGs directly via REST API or event/webhooks. Use Azure Logic Apps or Event Grid to detect events and trigger a Fabric Data Pipeline, which in turn can trigger your Airflow DAG.

If our response addressed, please mark it as Accept as solution and click Yes if you found it helpful.

Regards,
Chaithra.

[v-echaithra]

Hi @b_schmitt ,

Please follow this document inorder to set up Apache Airflow Jobs in Fabric.
Run a Fabric data pipeline and notebook using Apache Airflow DAG. - Microsoft Fabric | Microsoft Lea...

[b_schmitt]

I already followed the tutorial that you have linked. But since that describes a very basic DAG it does not answer any of my above questions.