Thoughts on separating Gen1/Gen2 Dataflows in Pro/Fabric Workspaces

[jeffshieldsdev]

I'm curious people's thoughts here:

I'm planning on keeping my Gen1 Ingestion Dataflows in a Pro Workspace then using Gen2 Dataflows in my Fabric Workspace to load into Lakehouse.

My thinking is Gen2 Dataflows reading from Gen1 Dataflows in Pro Workspace will use less Fabric CU(s) than if the Gen2 Dataflow just read from the source.

Ingesting from Snowflake could take 10 minutes, as Snowflake processes the query then delivers the results. In a Fabric Workspace this would be 10 minutes of CUs--why not keep this compute load in Shared Capacity?

I know this increases maintenance as I have Gen1 Dataflow: Source->Dataflow and Gen2 Dataflow: Gen1 Dataflow->Lakehouse, and latency as I have to orchestrate these refreshes--but it would save load on my Fabric Capacity.

[v-priyankata]

Hi @jeffshieldsdev 

Thank you for reaching out to Microsoft Fabric Community.

This is a valuable idea, and it deserves to reach more members of the community. Please consider marking your post as "Accepted as Solution" and giving it a "Kudos" to help others benefit from it as well.

Thanks again!

[v-priyankata]

Hi @jeffshieldsdev 

Thank you for reaching out to Microsoft Fabric Community.

This is Really smart idea!—thank you for sharing! The Gen1-to-Gen2 Dataflow strategy to offload ingestion and save on Fabric CUs is a smart.

Please accept your post as " Accept as solution " and give a "kudos" to assist other community members in resolving similar issues more efficiently.
Thank you.

[collinq · Accepted solution]

Hi @jeffshieldsdev ,

That sounds like a reasonable plan for me.  One of the thoughts I have is that Gen2 and Fabric will theoretically "grow" together in capability and we know that Microsoft is concentrating on the Fabric side. 

My big thought would be about functionality of the Gen1 and Gen2 based on your needs today.  Gen2 might be the lastest and greatest version but does not have all of the Gen1 capabilities TODAY, it may very well be that Gen2 is the more capable one in the future.