❗ Bug Report / Feature Request: Dynamic Dataflow Execution Fails When CI/CD is Enabled in Microsoft

 @variables('gold-dataflowid')

However, once CI/CD is enabled on the dataflow, the exact same pipeline setup fails, even though:

• The Dataflow ID is correct

• The workspace ID is properly set

• The pipeline logic is unchanged

• The activity is using the proper variable references

  The error returned is a generic failure message that provides no clear insight into the issue. Even manually executing the dataflow (with CI/CD enabled) using dynamic input fails in the same way — suggesting that the issue is tied directly to CI/CD behavior.

  ---

  🔍 What I’ve Observed:

  • ✅ Dynamic execution of dataflows via variables works when CI/CD is disabled

  • ❌ The same approach fails when CI/CD is enabled

  • ❌ Manual runs in CI/CD mode also fail

  • ⚠️ There is no clear error output, making this difficult to troubleshoot

    ---

    ✅ Why This Matters:

    Dynamic pipelines are a key part of reusable and scalable data engineering workflows. Not being able to trigger dataflows dynamically when CI/CD is enabled significantly limits:

    • Multi-environment support (dev/test/prod)

    • Modular pipeline design

    • Automation workflows in modern dataOps strategies

      ---

      🙏 Feature Ask / Bug Fix:

      Please consider:

      • Clarifying whether this behavior is expected or a limitation

      • Providing more descriptive error messages when dataflow execution fails under CI/CD

      • Enabling support for dynamic dataflow execution with CI/CD enabled, similar to how notebooks or other activities can be driven with runtime variables

        Let me know if anyone has encountered this issue or has found a workaround!

         

        Note: I used chatgpt only to paraphrase the context so people would better understand it. I didn't generate the whole idea out of chatgpt.

        Thanks,
        Charan Basireddy