Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

The Power BI Data Visualization World Championships is back! Get ahead of the game and start preparing now! Learn more

Reply
Lynk_RM
Helper I
Helper I

Efficient query to merge several SharePoint CSV files?

How can I make this query more efficient to cut down on load times? I have roughly 174 files (+1 daily) merged to provide 220k rows. The load times are becoming problematic. Please see the code below and help if possible:

 

 

 

let
    Source = SharePoint.Files("https://********.com/sites/******"),
    #"Filter Folder" = Table.SelectRows(Source, each ([Folder Path] = "https://***********.com/sites/****/***")),
    #"Filtered Rows" = Table.SelectRows(#"Filter Folder", each ([Extension] = ".csv")),
    #"Added Custom" = Table.AddColumn(#"Filtered Rows", "Custom", each Csv.Document([Content])),
    #"Removed Other Columns" = Table.SelectColumns(#"Added Custom",{"Custom"}),
    #"Expanded Custom" = Table.ExpandTableColumn(#"Removed Other Columns", "Custom", {"Column1", "Column11", "Column2", "Column3", "Column4", "Column6"}, {"Custom.Column1", "Custom.Column11", "Custom.Column2", "Custom.Column3", "Custom.Column4", "Custom.Column6"}),
    #"Promoted Headers" = Table.PromoteHeaders(#"Expanded Custom", [PromoteAllScalars=true]),
    #"Filtered Rows2" = Table.SelectRows(#"Promoted Headers", each ([Date] <> "Date")),
    #"Filter Null Dates" = Table.SelectRows(#"Filtered Rows2", each [Date] <> null and [Date] <> ""),
    #"Change Date Type" = Table.TransformColumnTypes(#"Filter Null Dates",{{"Date", type date}}),
in
    #"Change Date Type"

 

 

1 ACCEPTED SOLUTION
mahoneypat
Microsoft Employee
Microsoft Employee

That's a pretty straight forward query, so not much optimization potential.  One thing you could do is do your promote headers in the custom column step with the change below, which would eliminate that step and the Filter step to remove the rows with "Date".  You'll have to redo your Expand column steps with the new column names.

 

Replace  each Csv.Document([Content]) with  each Table.PromoteHeaders(Csv.Document[Content]))

 

That probably won't speed things up much.  Another thing to consider is to use that M code to create a Dataflow and then use that dataflow in your report instead.  That would move the refresh offline and your desktop file would refresh faster against the Dataflow.

 

Pat

 





Did I answer your question? Mark my post as a solution! Kudos are also appreciated!

To learn more about Power BI, follow me on Twitter or subscribe on YouTube.


@mahoneypa HoosierBI on YouTube


View solution in original post

1 REPLY 1
mahoneypat
Microsoft Employee
Microsoft Employee

That's a pretty straight forward query, so not much optimization potential.  One thing you could do is do your promote headers in the custom column step with the change below, which would eliminate that step and the Filter step to remove the rows with "Date".  You'll have to redo your Expand column steps with the new column names.

 

Replace  each Csv.Document([Content]) with  each Table.PromoteHeaders(Csv.Document[Content]))

 

That probably won't speed things up much.  Another thing to consider is to use that M code to create a Dataflow and then use that dataflow in your report instead.  That would move the refresh offline and your desktop file would refresh faster against the Dataflow.

 

Pat

 





Did I answer your question? Mark my post as a solution! Kudos are also appreciated!

To learn more about Power BI, follow me on Twitter or subscribe on YouTube.


@mahoneypa HoosierBI on YouTube


Helpful resources

Announcements
Power BI DataViz World Championships

Power BI Dataviz World Championships

The Power BI Data Visualization World Championships is back! Get ahead of the game and start preparing now!

November Power BI Update Carousel

Power BI Monthly Update - November 2025

Check out the November 2025 Power BI update to learn about new features.

FabCon Atlanta 2026 carousel

FabCon Atlanta 2026

Join us at FabCon Atlanta, March 16-20, for the ultimate Fabric, Power BI, AI and SQL community-led event. Save $200 with code FABCOMM.