- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
INSERT in fabric
How can I possibly go about performing INSERT in fabric.
To elaborate, I am querying sql db tbl with df gen2 connector and creating a Lakehouse tbl. The tbl contains a field called ID (integer). How can I only append only the new rows to the LH tbl based on bringing only the delta rows (max ID of sql tbl today - max ID of dataflow table yesterday)
Such as,
Select * from db tbl where ID > maxID dataflow yesterday and ID<= max ID sql tbl today
Is it possible to achieve? To be very clear I am not looking for a power bi incremental refresh approach as there is no datetime column.
To be very precise, I am asking if it is possible to run today, the yeterday's instance of dataflow table (contains all the IDs upto yeterday; i.e. dataflow instance) and compare it today's max ID in sql (sql instance) and query only the new IDs and append them to the lakehouse table (lakehouse instance) that contains the sql table instances upto yesterday (queried through dataflow). There is no UPDATE of rows required cause all the values of past IDs are not editable by the data entry operators.

- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @smpa01 ,
Thanks for using Fabric Community.
As I understand you are looking to implement incremental load using dataflow gen 2.
You can refer to this document inorder to get some idea: Pattern to incrementally amass data with Dataflow Gen2 - Microsoft Fabric | Microsoft Learn
Hope this is helpful. Please let me know if you have further queries.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
@Anonymous thanks for the link.
If you read through it, you will come across the below screenshot. What it is basically saying, query the full SQL table (select * from db tbl) and then utilize power query to pass on the MAX ID (from the exsitng dataflow) to filter only the rows that are in today's data set that are > yesterday's MAX ID. So if my db table has 1M+ rows , I need to send a query to sql db to query 1M+ rows everyday so that PQ can utilize filter to strain out only the delta rows (let's suppose 10 rows today) that I need to append to the lakehouse table. I want to send a query to sql db select * from db where ID > maxID dataflow yesterday and ID<= max ID sql tbl today) that can only return 10 rows without needing PQ to filter on the subsequent steps and available to me to append to the existing lakehouse table .
If I need to query the full sql table only to use filtering through PQ then it defeats the purpose. I should be able to author a custom SQL query with custom filters in place and send that query to SQL. I am writing native SQL query ( as oppose to transforming sql table using PQ ) and query folding is not avialble to me (and I can sacrifice that but not sending native sql query to db).
How do I resolve this?
Thank you in advance.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @smpa01 ,
Inorder to avoid query the full sql table, I suggest you create a new table latest_order_id (log table to store which is going to store the latest order_id till today). By this you can avoid quering entire table by simply fetching the order_id from other table.
Every time when you try to do an incremental load, try to update the existing value in latest_order_id table with latest order id from the latest data.
Hope it is helpful. Please let me know incase of further queries.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @smpa01 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others. Otherwise, will respond back with the more details and we will try to help .
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @smpa01 ,
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .
If you have any question relating to the current thread, please do let us know and we will try out best to help you.
In case if you have any other question on a different issue, we request you to open a new thread .
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Need more time pls

Helpful resources
Join us at the Microsoft Fabric Community Conference
March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount!
Fabric Monthly Update - February 2025
Check out the February 2025 Fabric update to learn about new features.

Subject | Author | Posted | |
---|---|---|---|
09-24-2024 04:08 AM | |||
01-28-2025 08:26 PM | |||
02-28-2025 06:48 AM | |||
07-15-2024 12:26 AM | |||
01-19-2025 09:23 PM |
User | Count |
---|---|
34 | |
17 | |
3 | |
3 | |
2 |
User | Count |
---|---|
40 | |
14 | |
13 | |
12 | |
10 |