Forum Discussion
Help! Is it possible to use Row_Number with Concurrent Inserts?
Hi arpost
1) Have you considered a control table for storing and managing that key value?
2) I agree that for concurrent load requirement, you will need additional logic to maintain and check for the high water mark value before the insert. If this is a batch process, you can load your data in parallel threads into a stage table, and then have a transform process and use the ROW_NUMBER() over to transform (built IDENTITY column) into the final table. This will be much more simplified process and more optimal throughput.
Please try this and let me know if the issue still persists.
Thanks.
Can you expound a bit more on both #1 and #2? Trying to think through how that'd work or solve the issue.
- Anonymous2 years agoNot applicable
Hi arpost
#1
I presume you are working on a Warehouse. Given that delta tables support transactions (Transactions in Warehouse tables - Microsoft Fabric | Microsoft Learn), you can use a (control) table just to persist the max key, using the strategy of your preferences.
Just a simple example: Lets suppose you have a table ControlTable with a column id. The following sp will update the id and return the new value atomically, therefore all the processes must invoke this sp to get their unique id.CREATE PROC [dbo].[GetMaxId] @currentID INT OUTPUT AS BEGIN -- Start a new transaction BEGIN TRANSACTION; -- Set the transaction isolation level to SERIALIZABLE SET TRANSACTION ISOLATION LEVEL SERIALIZABLE; -- Get the current ID SELECT @currentID = id FROM dbo.ControlTable; -- Increment the ID SET @currentID = @currentID + 1; -- Update the table UPDATE dbo.ControlTable SET id = @currentID; -- Commit the transaction COMMIT TRANSACTION; END
Another option would be using GUID instead of a number if the data type is not strictly necessary. You can get a unique identifier by running SELECT NEWID();
Hope this helps. Please let me know if you have any further questions.- arpost2 years ago
Post Prodigy
Is the use of the Serializable isolation level above supported in Fabric, though? Per this doc, only snapshot isolation level is supported; all others are ignored at time of query execution:
https://learn.microsoft.com/en-us/fabric/data-warehouse/transactions#transactional-capabilities
I've thought about GUIDs, but my concerns are two-fold:
- Millions of records will require millions of GUIDs.
- I could be wrong, but wouldn't GUIDs complicate things like drop tables and reinsert updated records, which is essential given that Fabric doesn't support ALTER TABLE?
- Anonymous2 years agoNot applicable
Hi arpost
The internal team has replied as follows:
Good catch... it appears that the customer is employing an OLTP-like process to populate the warehouse. While warehouses prioritize performance over integrity, it might be beneficial for them to reassess their hydration process and make it sequential to avoid these conflicts. Here is another example but I am afraid it may show the same issue under concurrent calls.
https://learn.microsoft.com/en-us/fabric/data-warehouse/generate-unique-identifiers
I would recommend the approach I mentioned earlier, in which the stored procedures can concurrently load into a table, and then CTAS (historical load)/INSERT(incremental) into the final table with the ROW_NUMBER.
Hope this helps.