Forum Discussion
Notebook vs Dataflow Gen 2
I would suggest you to stick with notebooks for your scenario.
Use Dataflow Gen 2 if....Sources/destinations are natively supported (OneLake, SharePoint, etc.)
The logic is simple (joins, filters, merges)
You want to expose transformations to citizen developers or analysts
If you need better orchestration or reuse then you have toconsider:
Wrapping your notebook logic in a pipeline
Using parameters and dynamic branching
Logging for better visibility
and yes you are right that dataflowgen2 is costly as it depends on the refresh frequency, size of data, and compute capacity it utilizes. It is tied more to capacity planning and licensing
On maintainablity,
- dataflowgen2 would be easier for citizen developers as it comes with simple ETL pipelines, readable steps, versioning via UI
notebook will be harder to onboard non developers but much more maintainable for advanced users if you plan to modularize & use clear functions with logging and comments
Please kudo and 'Mark' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.
I would suggest you to stick with notebooks for your scenario.
Use Dataflow Gen 2 if....
Sources/destinations are natively supported (OneLake, SharePoint, etc.)
The logic is simple (joins, filters, merges)
You want to expose transformations to citizen developers or analysts
If you need better orchestration or reuse then you have toconsider:
Wrapping your notebook logic in a pipeline
Using parameters and dynamic branching
Logging for better visibility
and yes you are right that dataflowgen2 is costly as it depends on the refresh frequency, size of data, and compute capacity it utilizes. It is tied more to capacity planning and licensing
On maintainablity,
- dataflowgen2 would be easier for citizen developers as it comes with simple ETL pipelines, readable steps, versioning via UI
notebook will be harder to onboard non developers but much more maintainable for advanced users if you plan to modularize & use clear functions with logging and comments
Please kudo and 'Mark' as answer if the reply was helpful. This will be benefitting other community members who face the same issue.