DataflowsStagingLakehouse / DataflowsStagingWarehouse in source control

[cw88]

Hello, 

i connected my workspace with Devops source control and i see that DataflowsStagingLakehouse / DataflowsStagingWarehouse is synced into the Devops repository. 

If i now push the repo objects into a new workspace, DataflowsStagingLakehouse / DataflowsStagingWarehous are created as "normal" lakehouse / warehouse. In addition to this source control shows me changes on the DataflowsStagingWarehouse ... ?

Does anyone know if the objects can be excluded from source control? Or other ideas?

Thanks!

[vivien57]

Hello,

I opened a post on a similar subject some time ago

https://community.fabric.microsoft.com/t5/General-Discussion/Deployment-Pipeline-Display-items-not-p...

Have a nice day,

Vivien

[Anonymous]

HI @cw88,

Did you means to use fabric contents within deployment pipeline?

If that is the case, I'd like to suggest you do these getting data operations on bottom level and only push the Lakehouse with result table to the top level.

If you want to know more about dataflow staging, you can take a look at the following document:

Dataflow Gen2 data destinations and managed settings - Microsoft Fabric | Microsoft Learn
Regards,

Xiaoxin Sheng

[cw88]

Hi @Anonymous ,

no, i use fabric content with the integrated source control with azure devops. The workspace items DataflowsStagingLakehouse & DataflowsStagingWarehouse getting detected from source control even though i don't see them in source workspace.

When i push the items from source control to a new workspace, the "staging" objects where also pushed as "normal" lakehouse / warehouse.... 😐

[Anonymous · Accepted solution]

Hi @cw88,

In fact, these hidden contents are generated to work with dataflow staging features. 

When staging is enabled on your queries (the default behavior), your data is loaded into the staging location, which is an internal Lakehouse only accessible by dataflows itself.

The feature usage is based on your data destinations.
If you are work with data Warehouse destination, you can't disable it becase staging is mandatory for data warehouse.
If you are work with lakehouse and other non-warehouse destinations, you can disable staging to directly written data to the destination.

When you not need to use these internal lakehouse any more, you can remove them from the workspace system storage.(they will auto generated if any other contents or features dependency this feature, so remove the dependency contents at first)

Regards,

Xiaoxin Sheng

[cwlu88]

Hello @Anonymous ,

i know the background from staging warehouse/lakehouse. The question is if it is possible to exclude this objects from source control? Because when the staging objects are committed to source control they get created as "visible objects" in any test/dev workspace.