Forum Discussion

cbsalling's avatar
cbsalling
Frequent Visitor
9 years ago

Timeouts with Power BI Service DirectQuery to HDInsight Spark Cluster

I’m trying to set up a connection from Azure Data Lake Store (ADLS) to Power BI through a HDInsight Spark Cluster. My exact data pipeline is as follows: CSV file with time series data stored in ADL...
Need Help
v-haibl-msft's avatar
v-haibl-msft
Microsoft Employee
9 years ago

cbsalling

 

Do you mean that you can load the same column data in Desktop but cannot in Service? How about the result if you apply a filter to reduce the row records returned in Service?

 

Best Regards,

Herbert

cbsalling's avatar
cbsalling
Frequent Visitor
9 years ago

Yes, essentially, although I believe that the data connection in the desktop client isn't live. It imports the data. Versus the cloud service, where it sets up a direct connection.

 

I haven't gotten around to trying filters or reducing the number of rows returned somehow. The data isn't in a great format - it's in a non-parseable date format with paired alphanumerical values. I can parse it via the query editor in the desktop. I have to use a language I'm not super familiar with if I want the data in a better format before I connect to it via the service. I was just curious if there was anyone out there who had actually worked with the HDInsight Spark data connector and what their experience was. 

 

I'll reshape the data before connecting to it and we'll see how it goes. Thanks, anyway!

 

Thanks,
Claire

  • v-haibl-msft's avatar
    v-haibl-msft
    Microsoft Employee
    9 years ago

    cbsalling

     

    Every action such as selecting a column or adding a filter will send a query back to the database. There are some tips to optimize your clusters for Power BI can be found here. And before selecting very large fields, consider choosing an appropriate visual type.

     

    Best Regards,

    Herbert