Forum Discussion
Adding new rows and adjusting original row data
- 1 year ago
I think this should work with the updated sample and requirements. Quick summary of changes:
-
Updated Position ID transformation in FixValTypes to handle -'s as blanks
-
Within Group, defined noOpen check and used in output step to just return grouped rows as is if no Open Positions are found in group
-
Within Group, updated amountFixed step to only apply Open Position Amount to Types "Position closed" and "corp action: Split"
-
Within Group, at newGen step (List.Generate):
-
Updated rows iteration to apply current Units to dividends (also, reorganized conditional for clarity)
-
Updated unit iteration to have the multiplied split value default to 1 (so, keep unit as is) in case split value is null (otherwise, dividend or any other non-split transactions with null split value would null out the unit getting passed forward)
-
let Source = Original_Updated, FixValsTypes = Table.TransformColumns( Source, { {"Date", each DateTime.From(_, "en-IN"), type datetime}, {"Amount", each Number.FromText(_, "en-US"), Currency.Type}, { "Units / Contracts", each if _ = "-" then null else Number.FromText(_, "en-US"), type number }, {"Realized Equity Change", each Number.FromText(_, "en-US"), Currency.Type}, { "Realized Equity", each Number.FromText(Text.Remove(_, " "), "en-IN"), Currency.Type }, {"Balance", each Number.FromText(Text.Remove(_, " "), "en-US"), Currency.Type}, {"Position ID", each if _ = "-" then null else Int64.From(_), Int64.Type}, {"NWA", each Number.FromText(_, "en-US"), Currency.Type} } ), SplitDetails = Table.SplitColumn( FixValsTypes, "Details", Splitter.SplitTextByEachDelimiter({"/", " "}), {"Details", "Currency", "Split details"} ), AddSplitValue = Table.AddColumn( SplitDetails, "Split value", each if [Split details] = null then null else [ split = Text.Split([Split details], ":"), output = Number.FromText(split{0}) / Number.FromText(split{1}) ][output], type number ), NewType = type table Type.ForRecord( Type.RecordFields(Type.TableRow(Value.Type(AddSplitValue))) & [ Previous Balance = [Type = Currency.Type, Optional = false] ], false ), PreGroupSort = Table.Sort( AddSplitValue, { {"Position ID", Order.Ascending}, {"Date", Order.Ascending} } ), Group = Table.Group( PreGroupSort, {"Position ID"}, { "Fix", each [ group = _, noOpen = not List.Contains( group[Type], "Open Position"), rowOrig = group{[Type = "Open Position"]}, unitOrig = rowOrig[#"Units / Contracts"], splitProduct = List.Product( List.Transform( Table.SelectRows( group, each [Type] = "corp action: Split" )[Split value], each 1 / _ ) ), unitFixed = unitOrig * splitProduct, amountOrig = rowOrig[Amount], amountFix = Table.ReplaceValue( group, each [Amount], each if List.Contains( {"Position closed", "corp action: Split"}, [Type]) then amountOrig else [Amount], Replacer.ReplaceValue, {"Amount"} ), rowCount = Table.RowCount(group), addPrevBalance = Table.Buffer( Table.FromColumns( Table.ToColumns(amountFix) & { {null} & List.RemoveLastN(group[Balance]) }, NewType ) ), firstRecordFixed = Record.TransformFields( Table.First(addPrevBalance), {{"Units / Contracts", each unitFixed ?? _}} ), newGen = List.Generate( () => [i = 0, rows = {firstRecordFixed}, unit = unitFixed], each [i] < rowCount, each let currentUnit = [unit], currentRow = addPrevBalance{[i] + 1} in [ i = [i] + 1, rows = [ splitSell = Record.TransformFields( currentRow, { {"Type", each "Split Sell"}, {"Units / Contracts", each currentUnit} } ), splitBuy = Record.TransformFields( currentRow, { {"Type", each "Split Buy"}, {"Units / Contracts", each unit} } ), dividend = Record.TransformFields( currentRow, {{"Units / Contracts", each currentUnit}} ), noChange = currentRow, output = if currentRow[Type] = "corp action: Split" then {splitSell, splitBuy} else if currentRow[Type] = "Dividend" then {dividend} else {noChange} ][output], unit = [unit] * (currentRow[Split value] ?? 1) ], each [rows] ), comboRows = List.Combine(newGen), output = if noOpen then group else Table.FromRecords(comboRows, NewType) ][output], NewType }, GroupKind.Local ), CombineGroups = Table.Combine(Group[Fix]), AddPricePQ = Table.AddColumn( CombineGroups, "PricePQ", each [Amount] / [#"Units / Contracts"], type number ) in AddPricePQ -
Hello MarkLaf and thanks to looking into this. From the quick look it works as expected. When applying the query to full data set, some new discoveries.
There are some transactions that do not have position ID. those are mostly balance deposits or withdrawals. Because of that, the step "Group" has null values in Position ID and query runs into error.
Other thing is that transactions may be of some other types like fees and dividends and for them, the no data (units or amounts) should be adjusted. those should have original values as in the source.
I've added some new lines to source data so you can simulate that on your side. If you could help me sort these out, i could test the whole data set and see if it works as expected.
Thanks again.
= Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("zZRLb9pAEMe/yopzsp6Z3dmHbxWkl1aiKs0J5YDAad1SjLATqd++s7Z5OURwqVQJacdm1r+Z/zzm8xGEDDAjIFRoc8M5hNHdaLotNupLVZdNWW3kefZtOv6UPc4mYjNoADkRdSAwBsXuX1kFETXa5MXaGzlDQI4Obfp71lTLX/Xe/eluPsJ4SrdwOz0dgVvEAe7RaxOTidp6OSO4QJ6suQgnzsD0cJdb4dsrcESnmbtvEB/RQVlnNVNyUUacQmuCdxjIYriIB5/Jr8Ob3GAOKeQ9WS3XVV2sBgF4pyMPtKf+XVAxWE1OTGdBY7yqfoogpAiMAt9GkPyW1W6rFssURK5m23XZnAahMEc4Zn5/KoJUpDNdAO3ibRL82wCudIDv248Tn/2NfPcuPxWgLb0lDe4/4N9aAJC7OXPXg5Nim7ow9RYkV/Xw+FWNd8WqbMaLXepJMkaz6/GHh7fG/WmEbcaUQeyBmCpuUrjT12K3Kb//aNRzUQxaHrQ95tmaLOMN7RSYoPGGOZcVx4kp6wly4hwTc1K+lqtiM5ww0mT3eXVmkNXku3EyxDqkEIgMIAfP8bKsrieSQsxJOjb5PYh+Spy36nNV13t1QHs3GtSRUerY7rRI2tlBhmeCgghKhxFioaV7H1Y/X+rmd7Fpen9ZiHhuehXYdVUSPSleqhbYDKFbUaKc9bnls2rtiueXVsHJolz/6W7a40d69TDqtv2c92/LNf44ORCf/gI=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Date = _t, Type = _t, Details = _t, Amount = _t, #"Units / Contracts" = _t, #"Realized Equity Change" = _t, #"Realized Equity" = _t, Balance = _t, #"Position ID" = _t, #"Asset type" = _t, NWA = _t])
So, to confirm the updated reqs:
1) Any Position ID groups (including where Position ID is null) that do not have an Open Position row included can just be left as is?
Example: no transformations are needed on the below outlined rows (1-2 & 6) as they have no associated Open Position?
2) For Position ID groups with an Open Position row:
a) ALL other rows should use the Open Position Amount?
Example: for the below 906... group, the Amount for ALL rows should match Open Position's, i.e. 50?
b) I'm guessing the Units should be based on most recent previous Units, whether that is Open Position or a split buy?
Example: If the timestamp of the Edit Stop Loss below put it after the first 1:10 split, then the associated Units should match the updated Units (which would be on the Split Buy row that gets added)?
c) Sort of covered already indirectly, but want to directly check, any row with a timestamp that is earlier than the Open Position row should also get assigned the Open Position row's (recalculated, if group has splits) Units? Same with Amount? Or should it stay as is?
Example: What should happen to Units and Amount of the Overnight fee row in the 906... group?
- bigk1 year agoHelper III
1) correct. But if the position has dividend, then the "Units" should be adjusted based on the previous transaction "Units" count. E.g. if dividend is paid after "Open Position", then "Units" count from that "Open Position" adjusted for total splits should be used. If Dividend was paid after split, then "Units" count from earlier spit should be taken as Units.
2a) No, only Open Position and Stock Splits should have the same amount as those are not changing the position worth. All other types of transactions should have their original amounts. However for dividends, "Units" should be adjusted as per point 1.
2b) Correct. But only for dividends and splits. Other transactions should have original units (usually they don't have any) and amounts.
2c) There should not be any other transaction with the same Position ID before the "Open Position" transaction. This is a sampling error from my side. I've generated new sample data which is more clear.
If you check Position ID 881596140, you will see that Position Closed has different amount than Position open and split. That is because it was closed with profit. "Position Closed" transactions should also not be anyhow changed and kept as is.
= Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("zVRNb9swDP0rQs6tTFKivm5D0tMGZFjWU9BDkLibtywOErfA/v0o2/lomjbZYcBOpGVK7+k9itPpAEIBWBAQKrTJcIIwuBmM1+VKfa63VVPVK/mefB0PPxb3k5HkDBpAImoMBMag5P2SVRBl2eYq1t5IDAE5OrT596Sp5z+3u/KHm+kA4zG6hevRcwjcQuzBPXptYk5RWy8xggvkyZqz4OALcD24SQYT5D3zerNWs3mGTmqyXlbNMQOFCeGAeVaDIDQs57vv4rsaEBdgehouWZHBXtAA0Wnm7gziA3RQ1lnNlEuUkaLQpuCdsLQY3lTBv1Jhh6zmy3pbLk4IeKcjn1yf+rWgYrCanKTOgsZ4UYDMIGQGRoFvGcDf+nB7LII0Rpe6ANrF6yT4twTeb0Ty/SvgjM/+Snz3Jn42oLXekgb3H+BfawDI3sTc9eCoXOcuzL0FuVTd3X9Rw025qJrhbJN7kozR7Hr4/cfr5PaYYXtjKiBmQFKA2XGT6Y6fy82q+va9UY9ledLyoO3hnm3KMmWgfQUmaLw8bkgmLWdMmZKQiBNmzFH1XC3K1ekLI012d68uDTIhffecDLEOcHm+uR6QFGIiadi8+U7kU1K8Vp/q7XYnDmjvBic2MoqN7UiLpJ09wXuhJ4ietH9BLGh534fFj6dt86tcNX29jGV8mXoV2HUmiZwUz5kFtkDozYJkfWqH6sGsTfn41Ao4mlXL391OezikFw+jbrvPeX/RrYc/", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Date = _t, Type = _t, Details = _t, Amount = _t, #"Units / Contracts" = _t, #"Realized Equity Change" = _t, #"Realized Equity" = _t, Balance = _t, #"Position ID" = _t, #"Asset type" = _t, NWA = _t])- MarkLaf1 year agoSuper User
I think this should work with the updated sample and requirements. Quick summary of changes:
-
Updated Position ID transformation in FixValTypes to handle -'s as blanks
-
Within Group, defined noOpen check and used in output step to just return grouped rows as is if no Open Positions are found in group
-
Within Group, updated amountFixed step to only apply Open Position Amount to Types "Position closed" and "corp action: Split"
-
Within Group, at newGen step (List.Generate):
-
Updated rows iteration to apply current Units to dividends (also, reorganized conditional for clarity)
-
Updated unit iteration to have the multiplied split value default to 1 (so, keep unit as is) in case split value is null (otherwise, dividend or any other non-split transactions with null split value would null out the unit getting passed forward)
-
let Source = Original_Updated, FixValsTypes = Table.TransformColumns( Source, { {"Date", each DateTime.From(_, "en-IN"), type datetime}, {"Amount", each Number.FromText(_, "en-US"), Currency.Type}, { "Units / Contracts", each if _ = "-" then null else Number.FromText(_, "en-US"), type number }, {"Realized Equity Change", each Number.FromText(_, "en-US"), Currency.Type}, { "Realized Equity", each Number.FromText(Text.Remove(_, " "), "en-IN"), Currency.Type }, {"Balance", each Number.FromText(Text.Remove(_, " "), "en-US"), Currency.Type}, {"Position ID", each if _ = "-" then null else Int64.From(_), Int64.Type}, {"NWA", each Number.FromText(_, "en-US"), Currency.Type} } ), SplitDetails = Table.SplitColumn( FixValsTypes, "Details", Splitter.SplitTextByEachDelimiter({"/", " "}), {"Details", "Currency", "Split details"} ), AddSplitValue = Table.AddColumn( SplitDetails, "Split value", each if [Split details] = null then null else [ split = Text.Split([Split details], ":"), output = Number.FromText(split{0}) / Number.FromText(split{1}) ][output], type number ), NewType = type table Type.ForRecord( Type.RecordFields(Type.TableRow(Value.Type(AddSplitValue))) & [ Previous Balance = [Type = Currency.Type, Optional = false] ], false ), PreGroupSort = Table.Sort( AddSplitValue, { {"Position ID", Order.Ascending}, {"Date", Order.Ascending} } ), Group = Table.Group( PreGroupSort, {"Position ID"}, { "Fix", each [ group = _, noOpen = not List.Contains( group[Type], "Open Position"), rowOrig = group{[Type = "Open Position"]}, unitOrig = rowOrig[#"Units / Contracts"], splitProduct = List.Product( List.Transform( Table.SelectRows( group, each [Type] = "corp action: Split" )[Split value], each 1 / _ ) ), unitFixed = unitOrig * splitProduct, amountOrig = rowOrig[Amount], amountFix = Table.ReplaceValue( group, each [Amount], each if List.Contains( {"Position closed", "corp action: Split"}, [Type]) then amountOrig else [Amount], Replacer.ReplaceValue, {"Amount"} ), rowCount = Table.RowCount(group), addPrevBalance = Table.Buffer( Table.FromColumns( Table.ToColumns(amountFix) & { {null} & List.RemoveLastN(group[Balance]) }, NewType ) ), firstRecordFixed = Record.TransformFields( Table.First(addPrevBalance), {{"Units / Contracts", each unitFixed ?? _}} ), newGen = List.Generate( () => [i = 0, rows = {firstRecordFixed}, unit = unitFixed], each [i] < rowCount, each let currentUnit = [unit], currentRow = addPrevBalance{[i] + 1} in [ i = [i] + 1, rows = [ splitSell = Record.TransformFields( currentRow, { {"Type", each "Split Sell"}, {"Units / Contracts", each currentUnit} } ), splitBuy = Record.TransformFields( currentRow, { {"Type", each "Split Buy"}, {"Units / Contracts", each unit} } ), dividend = Record.TransformFields( currentRow, {{"Units / Contracts", each currentUnit}} ), noChange = currentRow, output = if currentRow[Type] = "corp action: Split" then {splitSell, splitBuy} else if currentRow[Type] = "Dividend" then {dividend} else {noChange} ][output], unit = [unit] * (currentRow[Split value] ?? 1) ], each [rows] ), comboRows = List.Combine(newGen), output = if noOpen then group else Table.FromRecords(comboRows, NewType) ][output], NewType }, GroupKind.Local ), CombineGroups = Table.Combine(Group[Fix]), AddPricePQ = Table.AddColumn( CombineGroups, "PricePQ", each [Amount] / [#"Units / Contracts"], type number ) in AddPricePQ- bigk1 year agoHelper III
From the first look it seems that everything works as expected. Maybe some things will be spotted later, but as of now this case can be closed.
Thanks MarkLaf for yur help! You've saved me twice already including my other earlier request for help.
And thanks grazitti_sapna for your effort too!
PQ has limitless possibilities, unfortunately there is not enough time to learn everything. It is great to have such good community!
-