Forum Discussion
Post PartisanDealing with a huge csv file
I recently saw a post that one person claimed they had difficulties in loading a 12MB csv file into power bi desktop.
Their solution was to use dataflows - which I don't understand.
I don't quite understand why dataflows would be able to load the whole data as opposed to using the power query editor in power bi desktop.
another solution could be to split the csv into smaller chunks but this maximised the administrative efforts.
how would you go about dealing with a large csv file in power in desktop?
Thank you
I heard Dataflows is good for situations where one does not have permissions to access the database but another user doesalso called "breaking the security chain of custody" - I can not endorse that.
dataflow can be used as tool to reduce the semantic model size by trying to push much of the transformations away from the semantic model into the dataflowDataflows are glorified CSV files. Do these transforms at your own risk.
can a dataflow be used by a report in a different workspace?yes. Reusability is one of the positive features of dataflows. You will still need to have access to that "other" workspace.
14 Replies
- lbendlin
Super User
12MB is not larger for a CSV file. We casually ingest 800MB CSV files from SharePoint (and lots of them) and append these into the semantic model.
Keep in mind that dataflows are basically glorified CSV files in Azure Blob storage. So ingesting a CSV file in a dataflow is pointless 95% of the time.
The real question is about the performance of the data source. That's where dataflows can help - dataflows can shield the developer (NOT the report user) from a slow data source.
Consider using Binary.Buffer to prevent the data source from chunking.
mp390988The 800MB CV files you append to the semantic model, does it take you a lot of time? Are you able to do it without any issues using Power Bi desktop and selecting the TEXT/CSV connector? Do you use any special method other than the box standard import one would usually do using power bi desktop.
So when or under what scenario would dataflows be useful?
- lbendlin
Doesn't take long at all. You can assume that CSV and Parquet are the best performing formats for ingesting data from Import Mode sources.
We use the SharePoint Folder connector exclusively as we want to combine all the CSVs (in this case 50 CSVs at 800 MB each) into the semantic model.
So when or under what scenario would dataflows be useful?1. dataflows can shield the developer (NOT the report user) from a slow data source.
2. dataflows can be re-used in multiple semantic models.
- mp390988
When you say dataflow shields the developer (NOT the report user) from a slow data source - doesn't a slow data source also affect a report user as well? If so, why does a dataflow only applicable to developer and not report user?
- HarishKM
mp390988 Hey,
kindly follow the below steps
I will suggest you to upload your csv file to sharepoint folder of your team common folder.
Then you can use sharepoint folder.
You have to remove x/r from your sharepoint url - https://yourcompanyname.sharepoint.com/yourfoldername/then you can select or filter down your csv file then you are all set here.
Did I answer your question? Mark my post as a solution! and Give Kudos as well
Thanks
Harish M
