Difference between Spark and Databricks Dataflow

[joshua1990]

Hi experts!

I recently noticed that there is now a Databricks (Beta) connector for the Dataflows > Awesome!

In the past I always used Spark to connect to Databricks.

But now, is there any advantage to switch to the Databricks Connector?

[ibarrau]

Hi. I'm not sure if the connector is the same for dataflow than dataset. In my experience with datasets, the adventage is that the databricks connector is built for using databricks. That mean you have some advantages like query or turning on/off the cluster automatically when a refresh starts.

I would use the databricks one for those reasons. PowerBi will probably add updates and new features to the databricks connector for databricks and not the spark one.

I hope that helps,

[joshua1990]

Thanks a lot! What do you mean with Cluster in this conext?

[ibarrau · Accepted solution]

I mean the main processor at databricks. The engine you need to run in order to use it. The name it's cluster on a Data engineer or Science environemnt or you can also find it as SQL Endpoint or SQL Warehouse at a SQL environment.

Power Bi gets data from the metastore of databricks, you need .... "a cluster" turned on to read it.

I hope that make sense I can add pictures if it's still not clear for you.