Forum Discussion

Brianii's avatar
Brianii
Frequent Visitor
2 years ago

Is there a better solution to pulling data from CSV files located on SharePoint ?

I have data extractions in .CSV format saved to a SharePoint folder. Dataflows pull the data from the SharePoint folder and perform transformations. PowerBi then connects to dataflows, adds some DAX ...
Dataflow
Need Help
christinepayton's avatar
christinepayton
Icon for Most Valuable Professional rankMost Valuable Professional
2 years ago

Depends on your definition of efficient - what is the database source? Usually the way this is handled is using an ETL tool to put the data into some kind of SQL database. SharePoint will work in a pinch if you don't have a data engineer to integrate it in a data warehouse/mart/lake/whatever, but you'll start running into issues when you get into anything in the hundreds of thousands of rows or more. 

Brianii's avatar
Brianii
Frequent Visitor
2 years ago

The dataset sources are CSV extractions from our system saved in a SharePoint folder and then fed into Dataflows where transformations happen before connecting to PBIX.

 

My question is if this would be more effecient than the current set up. My resoning is that the Query Folding is not possible with CSV Files so if there's a Database between extractions and Dataflows it would speed up the process.
Current: Extraction > SharePoint Folder > Dataflows > PowerBi
Proposal: Extractions > SharePoint Folder > Database > Dataflows > PowerBI 

Could a Datamart function as a Database to feed Dataflows?