Forum Discussion
How to Use Dataflows Gen2 Without Creating a Lakehouse in Microsoft Fabric
Thanks for your help.
Hi bdpr_95 ,
Thanks for the details.
Yes, you are correct Since deployment pipelines support only Dataflow Gen2, you will need to go with Gen2 for Dev to Prod movement. But in Gen2, the staging Lakehouse is required and we cannot avoid it at the moment. Even if you only want to use the data in a semantic model, Fabric will still create or use a Lakehouse in the backend. So in your case, the best way is to use Gen2, let it load data into a Lakehouse you can keep it just for staging purpose, and then connect your semantic model to it. As of now, there is no option to achieve Gen1-like behavior without storage in Gen2.Hope this helps. Let me know if you need anything else.
Regards,
Community Support Team.
5 Replies
Hi bdpr_95,
Thanks for reaching out to the Fabric community.
In Dataflow Gen2, this is expected behavior. Unlike Gen1, Gen2 always needs a place to store the data while it is processing. That’s why Fabric automatically creates a staging Lakehouse if you don’t select one. As of now, there is no option to skip this Lakehouse step in Gen2.
If your requirement is to use dataflow only for transformation and directly use it in a semantic model without storing data, then Dataflow Gen1 is the right choice.
Otherwise, in Gen2 you can use the data from the Lakehouse output for your semantic model.
Kindly refer to the below documentation links for better understanding:
Data in Dataflow Gen2 staging - Microsoft Fabric | Microsoft Learn
Hope the above provided information help you resolve the issue, if you have any further concerns or queries, please feel free to reach out to us.
Regards,
Community Support Team.the problem here is, I have 2 environments, dev and prod. to do the deployment to previous stage (via deployment pipeline feature) dataflows gen1 cannot be moved, only dataflows gen2.
Hi bdpr_95 ,
Thanks for the details.
Yes, you are correct Since deployment pipelines support only Dataflow Gen2, you will need to go with Gen2 for Dev to Prod movement. But in Gen2, the staging Lakehouse is required and we cannot avoid it at the moment. Even if you only want to use the data in a semantic model, Fabric will still create or use a Lakehouse in the backend. So in your case, the best way is to use Gen2, let it load data into a Lakehouse you can keep it just for staging purpose, and then connect your semantic model to it. As of now, there is no option to achieve Gen1-like behavior without storage in Gen2.Hope this helps. Let me know if you need anything else.
Regards,
Community Support Team.
