The ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM.
Get registeredEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
Hello everyone,
I've been working on a Microsoft Fabric pipeline that dynamically triggers different activities (like notebooks or dataflows) based on parameters passed into it — specifically using item IDs to determine which path to execute.
Everything works perfectly when CI/CD is not enabled on the dataflow. I am able to pass in the Dataflow ID dynamically through a variable, like so:
@variables('gold-dataflowid')
However, once CI/CD is enabled on the dataflow, the exact same pipeline setup fails, even though:
The Dataflow ID is correct
The workspace ID is properly set
The pipeline logic is unchanged
The activity is using the proper variable references
The error returned is a generic failure message that provides no clear insight into the issue. Even manually executing the dataflow (with CI/CD enabled) using dynamic input fails in the same way — suggesting that the issue is tied directly to CI/CD behavior.
✅ Dynamic execution of dataflows via variables works when CI/CD is disabled
❌ The same approach fails when CI/CD is enabled
❌ Manual runs in CI/CD mode also fail
⚠️ There is no clear error output, making this difficult to troubleshoot
Dynamic pipelines are a key part of reusable and scalable data engineering workflows. Not being able to trigger dataflows dynamically when CI/CD is enabled significantly limits:
Multi-environment support (dev/test/prod)
Modular pipeline design
Automation workflows in modern dataOps strategies
Please consider:
Clarifying whether this behavior is expected or a limitation
Providing more descriptive error messages when dataflow execution fails under CI/CD
Enabling support for dynamic dataflow execution with CI/CD enabled, similar to how notebooks or other activities can be driven with runtime variables
Let me know if anyone has encountered this issue or has found a workaround!
Note: I used chatgpt only to paraphrase the context so people would better understand it. I didn't generate the whole idea out of chatgpt.
Thanks,
Charan Basireddy
You must be a registered user to add a comment. If you've already registered, sign in. Otherwise, register and sign in.