Forum Discussion
Improve performance to refresh data power query - multiples files
- 5 years ago
Hello twister8889
then your problem is not Power Query and your code but your infrastructure, and on this regard I cannot help you. You have to talk to your administrators.
Something about your code... I cannot see that you are creating your date-columns out of the filename but you create a static date column by using this syntax
#"Inserted Literal" = Table.AddColumn(#"Extracted Text Between Delimiters", "Date", each "01/01/2020", type text),You add a new column named "date" and adding a text-value. Then you are using the date-column for creating all your other columns like year etc. As edhans is also mentioning is that in a datamodel is better to avoid redundant data and this means that is better to create a date-dimension with all your data (like year) and then connect to your fact table through your date-column.
About your performance you can try this code... maybe it's faster
let Source = Folder.Files("C:\Users\User01\Desktop\Folder1\New Folder Performance"), #"Filtered Hidden Files1" = Table.SelectRows(Source, each [Attributes]?[Hidden]? <> true), #"Invoke Custom Function1" = Table.AddColumn(#"Filtered Hidden Files1", "Transform File from New Folder Performance", each #"Transform File from New Folder Performance"([Content])), #"Removed Other Columns" = Table.SelectColumns(#"Invoke Custom Function1",{"Name", "Transform File from New Folder Performance"}), #"Expanded Transform File from New Folder Performance" = Table.Buffer(Table.ExpandTableColumn(#"Removed Other Columns", "Transform File from New Folder Performance", {"Clients", "Stock", "Cash"}, {"Clients", "Stock", "Cash"})), #"Extracted Text Between Delimiters" = Table.TransformColumns(#"Expanded Transform File from New Folder Performance", {{"Name", each Text.BetweenDelimiters(_, "_", "_"), type text}}), #"Inserted Literal" = Table.AddColumn(#"Extracted Text Between Delimiters", "Date", each "01/01/2020", type text), #"Changed Type" = Table.TransformColumnTypes(#"Inserted Literal",{{"Date", type date}}), #"Inserted Month Name" = Table.AddColumn(#"Changed Type", "Month Name", each Date.MonthName([Date]), type text), #"Inserted Year" = Table.AddColumn(#"Inserted Month Name", "Year", each Date.Year([Date]), Int64.Type), #"Inserted Merged Column" = Table.AddColumn(#"Inserted Year", "Month", each Text.Combine({[Month Name], Text.From([Year], "pt-PT")}, "-"), type text), #"Inserted Day" = Table.AddColumn(#"Inserted Merged Column", "Day", each Date.Day([Date]), Int64.Type) in #"Inserted Day"
If this post helps or solves your problem, please mark it as solution (to help other users find useful content and to acknowledge the work of users that helped you)
Kudoes are nice too
Have fun
Jimmy
twister8889 You are putting everything in one big flat table. That isn't efficient for Power BI. Just have your date as a pure date in that table. Then have a date table that is separate that will have day, month, name, etc. May MANY different fields. By putting everything in one table is hugely inefficent. Power BI expects a Star Schema. Below is a code to my date table, and the importance of a star schema.
Microsoft Guidance on Importance of Star Schema
And to show you the importance of this, Amir Netz, the person largely responsible for the existence of Power BI in its current implementation, explains why flat tables are bad for Power BI.
Creating a Dynamic Date Table in Power Query
Doing these will both improve the performance of the DAX model itself and speed up the refresh time as you can get rid of all of thse Add Column steps and radically reduce the size of the data being loaded.
Hello, edhans thanks for your answer...
I will try to implement this solution, dim date. However, in my case, I think the problem is my structure. I will create this solution with two tables, dim date and fact file with a count of rows, and I go back here with feedback