Generate spark code from Dataflow Gen2

[fbcideas_migusr]

It is really cool that the visual query editor generates SQL.

If we could also generate spark code from Dataflow Gen2, we would be able to do glass-box data engineering.

[fbcideas_migusr]

In summary, it is the ability to convert data flow Gen2 into a spark notebook or Warehouse Query.

[nishalit]

Update 6/26/2024:

Thanks for the comment and context!

The way that the Visual Query editor works in Power BI Datamarts and Fabric Data Warehouse is by leveraging the query folding capabilities of the underlying connector for each of those data sources. This article might help understand a bit better how things get evaluated within Power Query and how some queries can leverage query folding to evaluate things at the data source. Nevertheless, this doesn't mean that all Power Query / M / Dataflow code can be translated to the underlying source and there are currently no plans to create such translation tool.

I'll leave this idea open so other folks can chime in and vote for it, but even today we don't have a full translation layer of all M code functionality to other languages.