Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
6 years ago
Solved

power query refresh really slow- excel file

Hi Everyone,

New to Power BI and Power Query here.

I'm trying to combine 10 excel files in a folder (each one is about 15 MB), when I apply changes, it took me like 4hours to refresh the query and used all of the memory and CPU!!

I've unchecked a few boxes in Settings to help speed up the load and read countless threads about buffer and folding but they don't seem to be applicable for me or I simply don't know how to use it.

 

Can anyone please take a look at my query and let me know how I can fix it?

 

Many Thanks!!!

 

let
Source=Folder.Files("M:\Dept\Global Interim Data Set (GIDS)"),
#"Removed Other Columns" = Table.SelectColumns(Source,{"Content", "Name"}),
#"Added Custom" = Table.AddColumn(#"Removed Other Columns", "Custom", each Excel.Workbook([Content])),
#"Removed Columns" = Table.RemoveColumns(#"Added Custom",{"Content"}),
#"Expanded Custom" = Table.ExpandTableColumn(#"Removed Columns", "Custom", {"Name", "Data", "Item", "Kind", "Hidden"}, {"Custom.Name", "Custom.Data", "Custom.Item", "Custom.Kind", "Custom.Hidden"}),
#"Filtered Rows1" = Table.SelectRows(#"Expanded Custom", each Text.Contains([Custom.Name], "Client") and not Text.Contains([Custom.Name], "$")),
#"Removed Columns1" = Table.RemoveColumns(#"Filtered Rows1",{"Custom.Name", "Custom.Kind", "Custom.Hidden"}),
#"obtain date" = Table.AddColumn(#"Removed Columns1","Date", each Text.Select([Name],{"0".."9"})),
#"Removed Columns2" = Table.RemoveColumns(#"obtain date",{"Name", "Custom.Item"}),
SkipRows = Table.AddColumn(#"Removed Columns2","Skip first 4",each Table.PromoteHeaders(Table.Skip([Custom.Data],4))),
#"Removed Columns4" = Table.RemoveColumns(SkipRows,{"Custom.Data"}),
unioncol = List.Union(
List.Buffer(
List.Transform(#"Removed Columns4"[Skip first 4], each Table.ColumnNames(_)))),
ExpandCol = Table.Buffer(
Table.ExpandTableColumn(#"Removed Columns4", "Skip first 4",unioncol)),
RemoveEmpty = FnRemoveEmptyColumns(ExpandCol),
ColName = Table.ColumnNames(RemoveEmpty), 

#"Removed Duplicates" = Table.Buffer(Table.Distinct(RemoveEmpty, List.Range(Table.ColumnNames(RemoveEmpty),1,162)))
in
#"Removed Duplicates"

  • Jimmy801's avatar
    Jimmy801
    6 years ago

    Hello Anonymous 

     

    I undestand. Then I can't figure out how these 4 lines of everey sheet are removed after you've combined them.

    I would go for

    - Create one query that threahts your file like you need it

    - Create a parameter with one path/name of a sample path

    - replace the filename in your first query with the parameter and convert it to a function

    - use the data source to extract the file names of this folder... in case do some filtering

    - add new column where you use your funciton and path and name as input.

     

    That should do the trick

     

    If this post helps or solves your problem, please mark it as solution.
    Kudos are nice to - thanks
    Have fun

    Jimmy

6 Replies

    • Anonymous's avatar
      Anonymous
      Not applicable

      Hi dax , 

       

      I read the post and converted my file to .csv that did decrease the refreshing time to ~3 hours (slightly increase). is there any other way to speed it up??? Maybe I'm asking for too much....

       

      • Jimmy801's avatar
        Jimmy801
        Community Champion

        Hello Anonymous 

         

        very difficult to get an idea what exactly happens here.

        - Columns were expanded and without use deleted afterwards,

        - by extracting a plain xlsx-file i don't get how a column reference can be a Table in this case for "Table.Skip" on row level - SkipRows = Table.AddColumn(#"Removed Columns2","Skip first 4",each Table.PromoteHeaders(Table.Skip([Custom.Data],4))),

        - a customfunction is used (not knowing what it does) - maybe use table.buffer in there

        - I don't know if Table.Buffer at the end is very usefull

         

        Maybe a change of concept in retrieving data would be best (maybe to create function to query the files, and put the data together at the end other change logic - difficult to answer if the real goal is not known.

         

        BR

        Jimmy