Forum Discussion
Not able to find Custom SQL option in Dataflow when source is databricks
Hi All,
I am trying to create a dataflow on top of databricks as data source. However, I am not able to find any option to write custom SQL. Though, it is there when I change my source to HANA or SQL Server.
Could you please help here.
Thanks,
Sarif
Hi mhmmd_srf
When creating a dataflow in Power BI using Databricks as the data source, you may notice that the option to write custom SQL queries is not available, unlike with traditional relational sources like SQL Server or SAP HANA. This limitation exists because the Databricks connector in Power BI dataflows currently does not support query folding or direct SQL input within the Power Query Online editor. Instead, it enforces a more GUI-based approach where you must use the standard table and column selection method followed by transformation steps using Power Query (M language).
This can be frustrating if you rely on complex SQL logic or want to optimize performance by pushing query logic down to Databricks. One workaround is to define your transformations as views or custom queries within Databricks itself, and then reference those views from Power BI dataflows. Another alternative is to use a Power BI Desktop file to connect to Databricks, write your custom SQL there, and then export the M script to use inside the dataflow (though this still won’t enable direct SQL execution, it helps preserve logic). Until Microsoft adds support for native SQL editing for Databricks in dataflows, these workarounds are the only viable approaches.
Hi mhmmd_srf ,
I hope the response provided by the super users have addressed the issue. If yes please please mark the helpful reply or share your solution and accept it as solution, it will be helpful for other members of the community who have similar problems as yours to solve it faster.
Regards,
Chaithra.
9 Replies
Hi mhmmd_srf
It is possible to write a custom sequel query using data bricks. Here is an example below on how to achieve this.
In the example below, you would need to replace the details with your databricks, plus the details and SQL code.
let Source = Databricks.Catalogs("adb-1234567890123456.7.azuredatabricks.net", "/sql/1.0/warehouses/abc123def456", [Catalog="samples", Schema="nyctaxi"]), Query = Value.NativeQuery(Source, "SELECT * FROM samples.nyctaxi.trips WHERE tpep_pickup_datetime >= DATEADD(day, -7, CURRENT_DATE)") in QuerySo, I need to do this in Advanced Editor. No option while creating connection like we have if data source is SQL Server?
Thanks,
MS
Hi mhmmd_srf
When creating a dataflow in Power BI using Databricks as the data source, you may notice that the option to write custom SQL queries is not available, unlike with traditional relational sources like SQL Server or SAP HANA. This limitation exists because the Databricks connector in Power BI dataflows currently does not support query folding or direct SQL input within the Power Query Online editor. Instead, it enforces a more GUI-based approach where you must use the standard table and column selection method followed by transformation steps using Power Query (M language).
This can be frustrating if you rely on complex SQL logic or want to optimize performance by pushing query logic down to Databricks. One workaround is to define your transformations as views or custom queries within Databricks itself, and then reference those views from Power BI dataflows. Another alternative is to use a Power BI Desktop file to connect to Databricks, write your custom SQL there, and then export the M script to use inside the dataflow (though this still won’t enable direct SQL execution, it helps preserve logic). Until Microsoft adds support for native SQL editing for Databricks in dataflows, these workarounds are the only viable approaches.
hi Poojara_D12 thanks.. this is helpful.
Hi mhmmd_srf ,
We wanted to kindly follow up to check if the solution provided for the issue worked? or Let us know if you need any further assistance?
If our response addressed, please mark it as Accept as solution and click Yes if you found it helpful.Regards,
Chaithra.Hi ,
Where is the option to mark as helpful? Can u plz guide me?
Thanks,
MS
Hi mhmmd_srf ,
May I ask if you have gotten this issue resolved?
If it is solved, please mark the helpful reply or share your solution and accept it as solution, it will be helpful for other members of the community who have similar problems as yours to solve it faster.
Regards,
Chaithra.Hi mhmmd_srf ,
I hope the response provided by the super users have addressed the issue. If yes please please mark the helpful reply or share your solution and accept it as solution, it will be helpful for other members of the community who have similar problems as yours to solve it faster.
Regards,
Chaithra.I just did.
