Forum Discussion
Dataflow gen2 - multiple staging artifacts
You're welcome. While I don't find any documentation that specifically confirms the reasoning, I believe the purpose for having multiple staging warehouses is because it will provide better capabilities for things like rollback and auditing, and possibly parallel processing as well. Since dataflow gen2 CICD is still in preview some things are likely to change, this is probably one of them. Please let me know if I can answer any other questions. Thanks!
Hello NiklasBDP - thanks for posting in the Fabric Community. Dataflow Gen2 uses staging artifacts, named "DataflowsStagingLakehouse" and "DataflowsStagingWarehouse," to store intermediate data during transformations, enhancing performance and reliability. These artifacts are automatically created and managed by Dataflow Gen2, hidden from users in the workspace but potentially visible in other tools, as you have seen. Since these are automatically created and managed by Dataflow Gen2, the only way you can truly control the creation of the staging artifacts is by disabling staging for the tables in the dataflow gen2. Please let me know if I can answer any other questions.
Thank you for the reply.
With a warehouse as destination I do not think I can disable staging as it is required.
With CI/CD enabled the dataflow calls them StagingWarehouseForDataflows instead of DataflowsStagingWarehouse and without CI/CD it works storing temp data from different dataflows in a single artifacts. However enabling CI/CD I start getting multiple staging artifacts (with date/time stamp) one for each dataflow which gets a bit messy. For like a month ago I was able to create multiple dataflows with CI/CD enabled storing data in a single staging artificat without (date/time) prefix, so I was wondering if I enabled something different.You're welcome. While I don't find any documentation that specifically confirms the reasoning, I believe the purpose for having multiple staging warehouses is because it will provide better capabilities for things like rollback and auditing, and possibly parallel processing as well. Since dataflow gen2 CICD is still in preview some things are likely to change, this is probably one of them. Please let me know if I can answer any other questions. Thanks!
