Forum Discussion
Lookup Formula in Power Query
- 2 years ago
If I understand you correctly, paste the code below into the Advanced Editor. It seems to work with your data sample.
Read the code and comments to better understand the algorithm.
let //change next line to reflect actual data source Source = Excel.CurrentWorkbook(){[Name="Table3"]}[Content], #"Changed Type" = Table.TransformColumnTypes(Source,{{"Parse", type date}}), //add index column to retain original order #"Added Index" = Table.AddIndexColumn(#"Changed Type", "Index", 0, 1, Int64.Type), //group by "Emp" #"Grouped Rows" = Table.Group(#"Added Index", {"Emp"}, { //For each emp {"all", (t)=> let //ensure date is sorted ascending sort = Table.Sort(t,{"Parse",Order.Ascending}), //shift segment down and up to compare this month to last month and next month shift = Table.FromColumns( Table.ToColumns(sort) & {{null} & List.RemoveLastN(t[Code Segment])} & {List.RemoveFirstN(t[Code Segment]) & {null}}, {"Parse","Emp","Code Segment","Index", "Shift Seg Down", "Shift Seg Up"}), #"Add SegName" = Table.AddColumn(shift,"Segment Name", each if [Code Segment] = ([Shift Seg Down]??[Code Segment]) then null else [Shift Seg Down]), #"Add Xfr Cnt" = Table.AddColumn(#"Add SegName","Transfer Count", each if [Code Segment] <> ([Shift Seg Up]??[Code Segment]) then -1 else if [Code Segment] = ([Shift Seg Down]??[Code Segment]) then 0 else 1), #"Remove Shifted" = Table.RemoveColumns(#"Add Xfr Cnt",{"Shift Seg Down","Shift Seg Up"}) in #"Remove Shifted", type table[Parse=date, Emp=text, Code Segment=text, Index=Int64.Type,Segment Name=text, Transfer Count=Int64.Type] }}), //Expand and sort the Grouped Columns #"Removed Columns" = Table.RemoveColumns(#"Grouped Rows",{"Emp"}), #"Expanded all" = Table.ExpandTableColumn(#"Removed Columns", "all", {"Parse", "Emp", "Code Segment", "Index", "Segment Name", "Transfer Count"}), #"Sorted Rows" = Table.Sort(#"Expanded all",{{"Index", Order.Ascending}}), #"Removed Columns1" = Table.RemoveColumns(#"Sorted Rows",{"Index"}) in #"Removed Columns1"
There is something other than the amount of data and/or the algorithm that is causing your slow execution.
I just set up a sample table in Excel with 10,000 rows.
I ran the query and it executed in a fraction of a second, returning all 10,000 rows.
I then ran it in Power BI, with the excel table as the data source. The execution time was measured at about three seconds, whether or not I had profiling set to the first 1000 or the entire data set.
Closing and saving it to Power BI showed that there were, indeed, 10,000 rows.
If I understand what you are doing, you are copying the information from the Power Query editor and pasting it into Excel. If that is the case, that is NOT a method to determine how many rows were loaded or processed.
- Yeah the last point was correct, I was copying the data from the Power Query Editor later i loaded it to PBI and i loaded all the rows.
- I also checked with the sample data of 500 thousand rows, yes that was loaded very quickly.
- For my data, I consolidating 4 tables using append queries and using only the appended table but i have created many columns and condition columns it loaded quickly but after this step it takes a lot of time to load in PQ and even in PBI.
- I dont know whether it is bug or not, please look at the attached screenshot:
- After i Click show errors:
- When i click cancel and check in PQ, there were no errors but everytime when i refresh this comes.
- ronrsnfld2 years agoSuper User
Sometimes that is due to a data type error. Do an internet search on the phrase "power query 1 of loaded queries contain errors, but I cannot view the error" and you'll see plenty of ideas. In addition to doing that, if you look at the individual queries, you may see a column that has a red top border -- that should be the column with the errors. You may be able to filter for the error and determine what is going on.
- Anonymous2 years agoNot applicable
Yeah thats the problem, the power query has some loading issues for this file, It takes forever to load and also sucks my memory and battery.