Dataflow gen2 - multiple staging artifacts

[NiklasBDP]

Hi,

If I chose to create a dataflow gen2 with CI/CD with warehouse as data destination it worked in the begining reusing StagingWarehouseForDataflows for multiple dataflows.

However recently I am getting multiple StagingWarehouseForDataflows with a data/time postfix like StagingWarehouseForDataflows_2025...... for each new dataflow, and when I create a new one StagingWarehouseForDataflows_2025........ (but they are hidden as attended from the workspace). However this is a bit messy when accessing the workspace from ADS, as they are not hidden here.
Is there some sort of setting that I have enabled or what happend, because I would like just one StagingWarehouseForDataFlows for all dataflows as I had in the beginning :)?

[TBNPieter]

Hey NiklasBDP,

I am seeing the exact same thing and it is quite annoying. We are implementing a Fabric architecture that relies very heavy on dataflows. We are currently testing for using CI/CD so are making DF2 with CI/CD and this just feels very messy to have multiple StagingWarehouseForDataflows. This has to be fixed somehow? Any news or did you find a solution to your problem?

Kind regard,s

[NiklasBDP]

No I did not find a solution except that jennratten, wrote that this is in preview and hopefully will be fixed before final release. So I do not enable CI/CD for dataflows at the moment.

[jennratten]

Hello @NiklasBDP - thanks for posting in the Fabric Community.  Dataflow Gen2 uses staging artifacts, named "DataflowsStagingLakehouse" and "DataflowsStagingWarehouse," to store intermediate data during transformations, enhancing performance and reliability. These artifacts are automatically created and managed by Dataflow Gen2, hidden from users in the workspace but potentially visible in other tools, as you have seen. Since these are automatically created and managed by Dataflow Gen2, the only way you can truly control the creation of the staging artifacts is by disabling staging for the tables in the dataflow gen2.  Please let me know if I can answer any other questions.

[NiklasBDP]

Thank you for the reply.

With a warehouse as destination I do not think I can disable staging as it is required.
With CI/CD enabled the dataflow calls them StagingWarehouseForDataflows instead of DataflowsStagingWarehouse and without CI/CD it works storing temp data from different dataflows in a single artifacts. However enabling CI/CD I start getting multiple staging artifacts (with date/time stamp) one for each dataflow which gets a bit messy. For like a month ago I was able to create multiple dataflows with CI/CD enabled storing data in a single staging artificat without (date/time) prefix, so I was wondering if I enabled something different. 

[jennratten · Accepted solution]

You're welcome. While I don't find any documentation that specifically confirms the reasoning, I believe the purpose for having multiple staging warehouses is because it will provide better capabilities for things like rollback and auditing, and possibly parallel processing as well.  Since dataflow gen2 CICD is still in preview some things are likely to change, this is probably one of them.  Please let me know if I can answer any other questions.  Thanks!