Forum Discussion
Pivot...UnPivot??? How to fix this bad data setup
Use this. That line was not needed.
let Source =Table.FromRows(Excel.CurrentWorkbook(){[Name="Table2"]}[Content]), Custom1 = [a = List.Skip(Table.ToColumns(Source)), b = {a{0}} & {a{3}} & {a{1}} & {List.Skip(a{1}) & {null}}, c = Table.FromColumns(b, {"Parent ID", "Event","Start Date", "End Date"}) ][c] in Custom1I think what yoy are asking for is result for a group of Parent ID. Then use this
let Source =Table.FromRows(Excel.CurrentWorkbook(){[Name="Table2"]}[Content]), Custom1 = Table.Combine(Table.Group(Source, {"ParentId"}, {"All", each [a = List.Skip(Table.ToColumns(_)), b = {a{0}} & {a{3}} & {a{1}} & {List.Skip(a{1}) & {null}}, c = Table.FromColumns(b, {"Parent ID", "Event","Start Date", "End Date"}) ][c]})[All]) in Custom1
I must be doing something wrong because I get a error of Token Idenfitie expected at the 2nd let. This is what I have:
let
Source =Table.FromRows(Excel.CurrentWorkbook(){[Name="Table2"]}[Content]),
let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [HxId = _t, ParentId = _t, CreatedDate = _t, OldValue = _t, NewValue = _t]),
Custom1 = [a = List.Skip(Table.ToColumns(Source)), b = {a{0}} & {a{3}} & {a{1}} & {List.Skip(a{1}) & {null}}, c = Table.FromColumns(b, {"Parent ID", "Event","Start Date", "End Date"}) ][c]
in
Custom1Use this. That line was not needed.
let
Source =Table.FromRows(Excel.CurrentWorkbook(){[Name="Table2"]}[Content]),
Custom1 = [a = List.Skip(Table.ToColumns(Source)), b = {a{0}} & {a{3}} & {a{1}} & {List.Skip(a{1}) & {null}}, c = Table.FromColumns(b, {"Parent ID", "Event","Start Date", "End Date"}) ][c]
in
Custom1
Oh wow. You may have just saved us hours of work. Thank you so very much for sharing.
After examining the data, I ran into issues. Unfortunately, the data is so messed up, I have null values in the new and old value columns. Sample:
Date OldValue NewValue 6/28/23 Submitted Incomplete 5/4/2024 Incomplete 6/10/2024 Withdrawn Is there a way to deal with that?
This is a very standard process with clear rules for identifying snapshot values
1. if there is no entry in the field history that means the current value is used
2. if there are entries in the field history before the snapshot date then use the New Value of the latest entry
3. if there are entries in the field history after the snapshot date then use the Old Value of the earliest entry
4. if the target field is blank then move on to the next extry
What is the output expected from the sample you just posted? Accordingly, I can change the code.
Hello,
I think I figured it out. Before I insert code you provided, I need to have the table sorted in in Hx Id and CreatedDate order.
Example:
This was the table straight from source:
After adding custom step - note the null value next to Response Required it should be 11/22/2023 12:00:00 AM, not null
After inserting a step to sort the data by ascending CreatedDate and Id, I get this after the Custom Step, which is the correct version of the history:
So thank you so much. All is well.
I really, really need help. Sorting the data by Id and CreatedDate did not help. I am getting very strange results. If I filter the data by a single ParentId before adding the code, then I get the expected results. But if I try to add the code to unfiltered data, something strange happens.
Sample Data:
Id ParentId CreatedDate OldValue NewValue 0178y00005LfiTpAAJ aAB8y0000004TjkGAE 07/12/2023 Incomplete Submitted 0178y00005cob9UAAQ aAB8y0000004TjkGAE 08/10/2023 Submitted PreApproved 0178y0000645cErAAI aAB8y0000004TjkGAE 10/04/2023 PreApproved Approved Expected Result:
Parent ID Event Start Date End Date aAB8y0000004TjkGAE Submitted 07/12/2023 08/10/2023 aAB8y0000004TjkGAE PreApproved 08/10/2023 10/04/2023 aAB8y0000004TjkGAE Approved 10/04/2023 null But here is what I get!
Parent ID Event Start Date End Date aAB8y0000004TjkGAE Submitted 07/12/2023 07/12/2023 aAB8y0000004TjkGAE PreApproved 08/10/2023 08/10/2023 aAB8y0000004TjkGAE Approved 10/04/2023 10/04/2023 Thank you so much for your help so far. But I really need more assistance. I have worked on this project for 2 weeks. I have to have something produced by Monday.
I think what yoy are asking for is result for a group of Parent ID. Then use this
let Source =Table.FromRows(Excel.CurrentWorkbook(){[Name="Table2"]}[Content]), Custom1 = Table.Combine(Table.Group(Source, {"ParentId"}, {"All", each [a = List.Skip(Table.ToColumns(_)), b = {a{0}} & {a{3}} & {a{1}} & {List.Skip(a{1}) & {null}}, c = Table.FromColumns(b, {"Parent ID", "Event","Start Date", "End Date"}) ][c]})[All]) in Custom1I really hate that I have to report this, but it is still not giving me the proper values. Example:
Id ParentId CreatedDate OldValue NewValue 0178y00006YNsBmAAL aAB8y0000000b6BGAQ 11/20/2023 Denied Submitted 0178y00006ZfHleAAF aAB8y0000000b6BGAQ 11/22/2023 Submitted Response Required 0178y00006ZrGuVAAV aAB8y0000000b6BGAQ 11/22/2023 Response Reqired Submitted 0178y00006bjYj5AAE aAB8y0000000b6BGAQ 11/27/2023 Submitted Approved Code I used (edited to filter data for all records with a specific ParentId to test results:
let Source = Access.Database(File.Contents("C:\Users\scastille539\AppData\NA_APP_TIMELINES\NA_App_Timeline.accdb"), [CreateNavigationProperties=true]), _tblNAHistory = Source{[Schema="",Item="tblNAHistory"]}[Data], #"Sorted Rows" = Table.Sort(_tblNAHistory,{{"Id", Order.Ascending}, {"CreatedDate", Order.Ascending}}), #"Removed Columns" = Table.RemoveColumns(#"Sorted Rows",{"CreatedById"}), #"Filtered Rows" = Table.SelectRows(#"Removed Columns", each [ParentId] = "aAB8y0000000b6BGAQ"), #"Sorted Rows1" = Table.Sort(#"Filtered Rows",{{"CreatedDate", Order.Ascending}, {"Id", Order.Ascending}}), Custom1 = Table.Combine(Table.Group(#"Sorted Rows1", {"ParentId"}, {"All", each [a = List.Skip(Table.ToColumns(_)), b = {a{0}} & {a{3}} & {a{1}} & {List.Skip(a{1}) & {null}}, c = Table.FromColumns(b, {"Parent ID", "Event","Start Date", "End Date"}) ][c]})[All]) in Custom1Result - it is completely wrong 😞
Parent Id Event Start Date End Date aAB8y0000000b6BGAQ Submitted 11/20/2023 11/27/2023 aAB8y0000000b6BGAQ Approved 11/27/2023 11/22/2023 aAB8y0000000b6BGAQ Submitted 11/22/2023 11/22/2023 aAB8y0000000b6BGAQ Response Required 11/22/2023 null My co-worker has created an incredibly complicated series of queries and vba code in MS Access. I wanted to avoid using it. But since I can't seem to get this to work, I have no choice.
Thank for trying to help. Your patience is commendable.
