Forum Discussion

mp390988's avatar
mp390988
Icon for Post Partisan rankPost Partisan
1 year ago
Solved

Dealing with a huge csv file

I recently saw a post that one person claimed they had difficulties in loading a 12MB csv file into power bi desktop.  Their solution was to use dataflows - which I don't understand.   I don't...
Need Help
  • lbendlin's avatar
    lbendlin
    1 year ago
    I heard Dataflows is good for situations where one does not have permissions to access the database but another user does

    also called "breaking the security chain of custody" - I can not endorse that.

     

    dataflow can be used as tool to reduce the semantic model size by trying to push much of the transformations away from the semantic model into the dataflow

    Dataflows are glorified CSV files. Do these transforms at your own risk.

     

    can a dataflow be used by a report in a different workspace?

    yes. Reusability is one of the positive features of dataflows.  You will still need to have access to that "other"  workspace.

lbendlin's avatar
lbendlin
Icon for Super User rankSuper User
1 year ago

12MB is not larger for a CSV file. We casually ingest 800MB CSV files from SharePoint (and lots of them) and append these into the semantic model.

 

Keep in mind that dataflows are basically glorified CSV files in Azure Blob storage.  So ingesting a CSV file in a dataflow is pointless 95% of the time.

 

The real question is about the performance of the data source. That's where dataflows can help - dataflows can shield the developer (NOT the report user) from a slow data source.

 

Consider using Binary.Buffer to prevent the data source from chunking.