Forum Discussion
Adding auto incremented identity column to deltalake table
Yes i am using Notebooks to Upserts / merge
And i already solved it by similar approach, just forgot to tell here.
My current solution was that i added the monotonically_increasing_id column to the delta tables at the bronze layer and added the current date of the data load for uniqiness of values then they update the silver delta tables like any other table with consistent schema between the two delta tables
Resolver IIIUsefGamal - Are you using Fabric Notebooks to do the upserts/merge? In the Fabric Notebooks you can create an auto-incrementing id; using a simliar pattern to what was provided for the data warehouse exmaple.
You can use a window w/monotonically_increasing_id() to create an auto-number function. Prior to the next data load, you would need to get the last number used for the new seed.
Then when merging your dataframes to prep for the update, you would autonumber the "new" rows and then take your seed value + rownumber to assign a key for the row number.
SImplified explanation above; but it is a process I've been using with success.
Yes i am using Notebooks to Upserts / merge
And i already solved it by similar approach, just forgot to tell here.
My current solution was that i added the monotonically_increasing_id column to the delta tables at the bronze layer and added the current date of the data load for uniqiness of values then they update the silver delta tables like any other table with consistent schema between the two delta tables