Forum Discussion
Adding new rows and adjusting original row data
- 1 year ago
I think this should work with the updated sample and requirements. Quick summary of changes:
-
Updated Position ID transformation in FixValTypes to handle -'s as blanks
-
Within Group, defined noOpen check and used in output step to just return grouped rows as is if no Open Positions are found in group
-
Within Group, updated amountFixed step to only apply Open Position Amount to Types "Position closed" and "corp action: Split"
-
Within Group, at newGen step (List.Generate):
-
Updated rows iteration to apply current Units to dividends (also, reorganized conditional for clarity)
-
Updated unit iteration to have the multiplied split value default to 1 (so, keep unit as is) in case split value is null (otherwise, dividend or any other non-split transactions with null split value would null out the unit getting passed forward)
-
let Source = Original_Updated, FixValsTypes = Table.TransformColumns( Source, { {"Date", each DateTime.From(_, "en-IN"), type datetime}, {"Amount", each Number.FromText(_, "en-US"), Currency.Type}, { "Units / Contracts", each if _ = "-" then null else Number.FromText(_, "en-US"), type number }, {"Realized Equity Change", each Number.FromText(_, "en-US"), Currency.Type}, { "Realized Equity", each Number.FromText(Text.Remove(_, " "), "en-IN"), Currency.Type }, {"Balance", each Number.FromText(Text.Remove(_, " "), "en-US"), Currency.Type}, {"Position ID", each if _ = "-" then null else Int64.From(_), Int64.Type}, {"NWA", each Number.FromText(_, "en-US"), Currency.Type} } ), SplitDetails = Table.SplitColumn( FixValsTypes, "Details", Splitter.SplitTextByEachDelimiter({"/", " "}), {"Details", "Currency", "Split details"} ), AddSplitValue = Table.AddColumn( SplitDetails, "Split value", each if [Split details] = null then null else [ split = Text.Split([Split details], ":"), output = Number.FromText(split{0}) / Number.FromText(split{1}) ][output], type number ), NewType = type table Type.ForRecord( Type.RecordFields(Type.TableRow(Value.Type(AddSplitValue))) & [ Previous Balance = [Type = Currency.Type, Optional = false] ], false ), PreGroupSort = Table.Sort( AddSplitValue, { {"Position ID", Order.Ascending}, {"Date", Order.Ascending} } ), Group = Table.Group( PreGroupSort, {"Position ID"}, { "Fix", each [ group = _, noOpen = not List.Contains( group[Type], "Open Position"), rowOrig = group{[Type = "Open Position"]}, unitOrig = rowOrig[#"Units / Contracts"], splitProduct = List.Product( List.Transform( Table.SelectRows( group, each [Type] = "corp action: Split" )[Split value], each 1 / _ ) ), unitFixed = unitOrig * splitProduct, amountOrig = rowOrig[Amount], amountFix = Table.ReplaceValue( group, each [Amount], each if List.Contains( {"Position closed", "corp action: Split"}, [Type]) then amountOrig else [Amount], Replacer.ReplaceValue, {"Amount"} ), rowCount = Table.RowCount(group), addPrevBalance = Table.Buffer( Table.FromColumns( Table.ToColumns(amountFix) & { {null} & List.RemoveLastN(group[Balance]) }, NewType ) ), firstRecordFixed = Record.TransformFields( Table.First(addPrevBalance), {{"Units / Contracts", each unitFixed ?? _}} ), newGen = List.Generate( () => [i = 0, rows = {firstRecordFixed}, unit = unitFixed], each [i] < rowCount, each let currentUnit = [unit], currentRow = addPrevBalance{[i] + 1} in [ i = [i] + 1, rows = [ splitSell = Record.TransformFields( currentRow, { {"Type", each "Split Sell"}, {"Units / Contracts", each currentUnit} } ), splitBuy = Record.TransformFields( currentRow, { {"Type", each "Split Buy"}, {"Units / Contracts", each unit} } ), dividend = Record.TransformFields( currentRow, {{"Units / Contracts", each currentUnit}} ), noChange = currentRow, output = if currentRow[Type] = "corp action: Split" then {splitSell, splitBuy} else if currentRow[Type] = "Dividend" then {dividend} else {noChange} ][output], unit = [unit] * (currentRow[Split value] ?? 1) ], each [rows] ), comboRows = List.Combine(newGen), output = if noOpen then group else Table.FromRecords(comboRows, NewType) ][output], NewType }, GroupKind.Local ), CombineGroups = Table.Combine(Group[Fix]), AddPricePQ = Table.AddColumn( CombineGroups, "PricePQ", each [Amount] / [#"Units / Contracts"], type number ) in AddPricePQ -
Hello grazitti_sapna and thanks for looking into this. I have 2 remarks. Adjusted open units were wrongly calculated but after changing sign from miltiply to dividsion, the adjusted open units were ok. Not sure how this will affect next steps as i could not proceed further due to the error in FlattenedSynthetic stepwhich gives an an error
Expression.Error: We cannot convert a value of type Record to type Table.
Hi bigk,
Replace this Part
if unitsBefore <> null and unitsAfter <> null then {
Record.Combine({commonFields, [Type="Synthetic Split - Reverse", Units=-unitsBefore, Amount=null]}),
Record.Combine({commonFields, [Type="Synthetic Split - Apply", Units=unitsAfter, Amount=null]})
} else {}
with
if unitsBefore <> null and unitsAfter <> null then {
{
Record.Combine({commonFields, [Type="Synthetic Split - Reverse", Units=-unitsBefore, Amount=null]}),
Record.Combine({commonFields, [Type="Synthetic Split - Apply", Units=unitsAfter, Amount=null]})
}
} else
{}
- bigk1 year agoHelper III
Thanks, but the issue still exist. Expression.Error: We cannot convert a value of type List to type Table.
Here is my full query. I've replaced first row to reference my original query that ends with AddedPrice.let
// Step 1: Load source data (already processed to AddedPrice)
Source = #"Transactions SIMULATION",// Step 2: Extract Split rows and compute split ratios
SplitRows = Table.SelectRows(Source, each Text.Contains([Type], "Split")),
WithRatios = Table.AddColumn(SplitRows, "SplitRatio", each
let
raw = Text.Trim([Split details]),
parts = Text.Split(raw, ":"),
numerator = try Number.FromText(parts{0}) otherwise null,
denominator = try Number.FromText(parts{1}) otherwise null
in
if numerator <> null and denominator <> null then numerator / denominator else null,
type number),// Step 3: Aggregate cumulative split ratio per Position ID
GroupedRatios = Table.Group(WithRatios, {"Position ID"}, {
{"TotalSplitFactor", each List.Product(List.RemoveNulls([SplitRatio])), type number}
}),// Step 4: Join back to source to adjust Open Position rows
JoinedToMain = Table.NestedJoin(Source, {"Position ID"}, GroupedRatios, {"Position ID"}, "SplitFactor", JoinKind.LeftOuter),
ExpandedFactors = Table.ExpandTableColumn(JoinedToMain, "SplitFactor", {"TotalSplitFactor"}),
AdjustedOpen = Table.AddColumn(ExpandedFactors, "AdjustedUnits", each
if [Type] = "Open Position" and [TotalSplitFactor] <> null then
[Units] * [TotalSplitFactor]
else [Units],
type number),// Step 5: Create synthetic rows for each split row
GeneratedSplits = Table.AddColumn(WithRatios, "SyntheticRows", each
let
originalRow = _,
positionID = originalRow[Position ID],
splitRatio = originalRow[SplitRatio],
date = originalRow[Transaction date],
baseRows = Table.SelectRows(Source, each [Position ID] = positionID and [Transaction date] < date and [Type] = "Open Position"),
unitsBefore = try baseRows{0}[Units] otherwise null,
unitsAfter = try unitsBefore * splitRatio otherwise null,
commonFields = Record.RemoveFields(originalRow, {"Type", "Units", "Amount"})
inif unitsBefore <> null and unitsAfter <> null then {
{
Record.Combine({commonFields, [Type="Synthetic Split - Reverse", Units=-unitsBefore, Amount=null]}),
Record.Combine({commonFields, [Type="Synthetic Split - Apply", Units=unitsAfter, Amount=null]})
}
} else
{}
),
FlattenedSynthetic = Table.Combine(List.Combine(GeneratedSplits[SyntheticRows])),// Step 6: Combine and sort all rows
CombinedAll = Table.Combine({AdjustedOpen, FlattenedSynthetic}),
SortedFinal = Table.Sort(CombinedAll, {{"Transaction date", Order.Ascending}}),// Step 7: Re-indexing and balance tracking
Reindexed = Table.AddIndexColumn(SortedFinal, "Index", 0, 1, Int64.Type),
WithIndex1 = Table.AddIndexColumn(Reindexed, "Index.1", 1, 1, Int64.Type),
MergedPrevBalance = Table.NestedJoin(WithIndex1, {"Index"}, WithIndex1, {"Index.1"}, "PrevRow", JoinKind.LeftOuter),
ExpandedPrevBalance = Table.ExpandTableColumn(MergedPrevBalance, "PrevRow", {"Balance"}, {"Previous blance"})in
ExpandedPrevBalance- bigk1 year agoHelper III
With the help of ChatGPT i've managed to get it working by modifying "FlattenedSynthetic" step to this:
Table.Combine(
List.Transform(
GeneratedSplits[SyntheticRows],
each Table.FromRecords(_)
)
)
However the issue now is that units are wrongly calcualted to split positions. First of all, units sould be multiplied by split ratio and not divided. But still other problem exists.The GeneratedSplits steps uses [Units] from the source data for new transactions. However, split transactions in the source do not have units. they are null. Also negative synthetic rows seems the replicate value of split rows, however they should adjust the previous row and not the current one. For example first negative slit should adjust the "Open Position" units. But the second negative split should adjust the first split new units and so on.