Problems with Databricks connector

[smaduri]

Hello Team,

We are building power bi reports and using azure Databricks clusters as our data source, we are using Spark (Beta) to get data into Power BI as suggested here (https://docs.azuredatabricks.net/user-guide/bi/power-bi.html) , and  we are using import mode in data connectivity.

We are working with 2 different datasets (e.g. dataset1 & dataset2) and using same type of connector , also same data connectivity mode(import mode) in both of the data sets.

Following are our observations :

1. Published a Power BI data set (e.g. dataset1) to Power BI Embed (A4 capacity) workspace and getting different random errors every time when we refresh Power BI data set. (Following are 3 different error messages in 3 refreshes), we also have gateway to connect to source.
1. Underlying error message: ODBC: ERROR [HY000] [DSI] The error message HardyHiveError could not be found in the en-US locale. Check that C:\Program Files\On-premises data gateway\ODBC Drivers\Simba Spark ODBC Driver\en-US\SparkODBCMessages.xml exists.
2. Data source error: ODBC: ERROR [08S01] [Microsoft][Hardy] (71) Failed to establish connection with unknown error.. The exception was raised by the IDbCommand interface
3. Data source error: The operation was throttled by the Power BI Premium because the operation was unable to reserve enough memory. Please try again later

1. Published the same Power BI data set(e.g. dataset1) to Power BI Pro workspace (default workspace that comes with MSIT.powerbi.com tenant for internal Microsoft users)
1. Dataset refresh is successful if we use gateway (though we really doesn’t require gateway for this data source connection).
2. Published another Power BI data set (e.g. dataset2) to Power BI Embed (A4 capacity)
1. This Power BI data set is refreshing successfully without gateway all the times.

We have couple of questions here :

1. Do we really need to have connection gateway for Spark(Beta) connector when we are using with Databricks clusters (we understand that we don’t need gateway for this)?
2. What are potential causes/reasons that another data set (dataset2) published to Power BI Embed workspace (A4 capactiy) is successfully refreshed.
3. Also, why we are seeing different refresh behaviors with Power BI embed workspace(A4 capacity) vs. Power BI Pro workspace (default MSIT.Powerbi.com workspace)
4. How do we achieve consistency in our data refresh process for all of our datasets if we want to make it productionalize these data sets.

Could you please help us to understand these scenarios.

Thanks in advance.

[v-jiascu-msft]

Hi @smaduri,

1. Since the Databricks clusters is a cloud service, the Data Gateway seems unnecessary. 

2. This shouldn't be a problem. Did you use the Workspace Version 2?

3. 4 Regarding refresh, as far as I know, there aren't any differences between the default workspace and App workspace. 

Best Regards,

Dale

[smaduri]

What is mean by workspace version 2, does it same as app workspace ?( if it is same as app workspace , then we have used app workspace only).

And we are clearly having issues with our data set refreshes, dataset2 is refreshing fine, and dataset1 is having issues as I mentioned above in the thread.

e.g. ODBC: ERROR [08S01] [Microsoft][Hardy] (71) Failed to establish connection with unknown error.. The exception was raised by the IDbCommand interface

Are there any other issues since Spark is beta connector, and do we need to consider any other factors since we are using Power BI embed (A4 capacity ) workspace

Thanks,

[v-jiascu-msft]

Hi @smaduri,

I would suggest you create a support ticket here. 

Best Regards,

Dale