Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Enhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.

❗ Bug Report / Feature Request: Dynamic Dataflow Execution Fails When CI/CD is Enabled in Microsoft

 

Hello everyone,

I've been working on a Microsoft Fabric pipeline that dynamically triggers different activities (like notebooks or dataflows) based on parameters passed into it — specifically using item IDs to determine which path to execute.

Everything works perfectly when CI/CD is not enabled on the dataflow. I am able to pass in the Dataflow ID dynamically through a variable, like so:

 

 @variables('gold-dataflowid')

However, once CI/CD is enabled on the dataflow, the exact same pipeline setup fails, even though:

  • The Dataflow ID is correct

  • The workspace ID is properly set

  • The pipeline logic is unchanged

  • The activity is using the proper variable references

    The error returned is a generic failure message that provides no clear insight into the issue. Even manually executing the dataflow (with CI/CD enabled) using dynamic input fails in the same way — suggesting that the issue is tied directly to CI/CD behavior.


    🔍 What I’ve Observed:

    • Dynamic execution of dataflows via variables works when CI/CD is disabled

    • The same approach fails when CI/CD is enabled

    • Manual runs in CI/CD mode also fail

    • ⚠️ There is no clear error output, making this difficult to troubleshoot


      Why This Matters:

      Dynamic pipelines are a key part of reusable and scalable data engineering workflows. Not being able to trigger dataflows dynamically when CI/CD is enabled significantly limits:

      • Multi-environment support (dev/test/prod)

      • Modular pipeline design

      • Automation workflows in modern dataOps strategies


        🙏 Feature Ask / Bug Fix:

        Please consider:

        • Clarifying whether this behavior is expected or a limitation

        • Providing more descriptive error messages when dataflow execution fails under CI/CD

        • Enabling support for dynamic dataflow execution with CI/CD enabled, similar to how notebooks or other activities can be driven with runtime variables

          Let me know if anyone has encountered this issue or has found a workaround!

           

          Note: I used chatgpt only to paraphrase the context so people would better understand it. I didn't generate the whole idea out of chatgpt.

          Thanks,
          Charan Basireddy

Status: New
Comments
4019Charan
Frequent Visitor
Hello @jennratten Could you vote the idea and support me so fabric would rectify the issue. As my current pipeline has dataflow activities I am not able to use dataflow with CI/CD enabled.