Forum Discussion

Regular Visitor

10 years ago

Spark+Hdinsight+1 mill rows=ODBC error

Hi, I am trying to load data from Spark. The csv data file is about 50Mb, and results in a table of one million rows. Not a very large data set and I have no problem to load it directly as csv. Ho...

Icon for Community Champion rank

Community Champion

10 years ago

afuyo File size limit for power bi desktop file is 1GB. I assume you're using Azure HDInsight Spark connector from power bi desktop which is still in beta and that is why i guess you're experiencing data size limit issue. You can get data from Spark on HDInsight straight into powerbi via powerbi service. Check out this post here. Maybe there is no limit via power bi service.

arify
Microsoft Employee
10 years ago
Hi afuyo, what version of Power BI are you using? We've fixed a similar issue in the latest versions.
- afuyo
  Regular Visitor
  10 years ago
  @ankitpatira Thank you for your input. Yes, indeed powerbi service seems to handle much larger data sets. I've tested with 12 GB and it worked fine with decent response times. However the limitations of power bi service, like not beeing able to create data model makes it too simplistic for our use. Thanks again for you answer and the link you've provided.
  
  arify Version: 2.38.4491.282 64-bit (August, 2016). I think it's thelatest version. I realize that Spark connector is still in beta. It will be exciting to see what path the development of the power bi program will take.
  - arify
    Microsoft Employee
    10 years ago
    That's odd, can you please reproduce this error while Tracing is enabled (Options > Diagnostics > Enable Tracing), and send us a frown email with those trace files (Microsoft.Mashup***.log) attached?
    
    (To send a frown, click the smiley at the top of the Power BI Desktop window)
    
    Thanks

Related content