Forum Discussion

angela_n's avatar
angela_n
Frequent Visitor
1 year ago
Solved

Using Terraform with Fabric and Airflow

Hey everyone,
I’m using Terraform to set up my fabric elements, and I want to use Airflow for orchestration instead of pipelines. I know Terraform is still in the early stages, but I think it’s going to be super beneficial in the long run.
Does anyone know how I can upload the DAG definitions to Fabric?

Thanks,

 

 

data engineering
  • angela_n's avatar
    angela_n
    1 year ago

    Hi Pallavi, 

     

    I think I get what you mean 😉. I did not see this config before 

    Thanks for your help. 

6 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi angela_n ,
    Thank you for reaching out to us on Microsoft Fabric Community Forum!

    While Terraform is great for provisioning Fabric resources, uploading Airflow DAGs directly via Terraform isn’t supported yet. DAGs need to be placed in the dags/ folder manually or through a CI/CD pipeline after setup. A common approach is to use Terraform for infrastructure and a separate script or pipeline to upload DAGs. This would be a great feature to have, so we recommend submitting it to the Fabric Ideas forum.
    Feedback submitted through these channels is frequently reviewed by the product teams and can contribute to meaningful improvements.

    Hope this helps.If so,consider accepting it as solution.

    Regards,
    Pallavi.

    • aterry_sit's avatar
      aterry_sit
      New Member
      1 year ago

      Hi, Im also trying to deploy DAGs to Airflow in Fabric, how would you deploy them through CI/CD? It doesnt looks like any Fabric CLI supports uplodaing DAG files at all and the storage account that Fabric uses doesnt seem to be exposed?

      • angela_n's avatar
        angela_n
        Frequent Visitor
        1 year ago

        Hi aterry_sit , we ended up installing our instance of airflow, outside fabric, and we update the DAGs using Azure Devops pipelines. Airflow is not production ready app within fabric

  • angela_n's avatar
    angela_n
    Frequent Visitor
    1 year ago

    Hi Pallavi,

    I get that uploading Airflow DAGs directly via Terraform isn’t supported yet and that they need to be placed in the dags/ folder manually. Could you give me some tips on how to set up a CI/CD pipeline for this, right now is showing as 'Unsupported'? 

     

    thanks, 

  • angela_n's avatar
    angela_n
    Frequent Visitor
    1 year ago

    Hi Pallavi, 

     

    I think I get what you mean 😉. I did not see this config before 

    Thanks for your help. 

    • Anonymous's avatar
      Anonymous
      Not applicable
      1 year ago

      Hi angela_n ,
      I hope the answer was helpful,If the issue has been resolved, please mark the helpful reply as a "solution" to indicate that the question has been answered and to assist others in the community.

      Thank you for your cooperation. Have a great day.