Forum Discussion
Dataflow gen2 - multiple staging artifacts
You're welcome. While I don't find any documentation that specifically confirms the reasoning, I believe the purpose for having multiple staging warehouses is because it will provide better capabilities for things like rollback and auditing, and possibly parallel processing as well. Since dataflow gen2 CICD is still in preview some things are likely to change, this is probably one of them. Please let me know if I can answer any other questions. Thanks!
Thank you for the reply.
With a warehouse as destination I do not think I can disable staging as it is required.
With CI/CD enabled the dataflow calls them StagingWarehouseForDataflows instead of DataflowsStagingWarehouse and without CI/CD it works storing temp data from different dataflows in a single artifacts. However enabling CI/CD I start getting multiple staging artifacts (with date/time stamp) one for each dataflow which gets a bit messy. For like a month ago I was able to create multiple dataflows with CI/CD enabled storing data in a single staging artificat without (date/time) prefix, so I was wondering if I enabled something different.
You're welcome. While I don't find any documentation that specifically confirms the reasoning, I believe the purpose for having multiple staging warehouses is because it will provide better capabilities for things like rollback and auditing, and possibly parallel processing as well. Since dataflow gen2 CICD is still in preview some things are likely to change, this is probably one of them. Please let me know if I can answer any other questions. Thanks!