Implementing Incremental Ingestion in Fabric Lakehouse from an Oracle Table with Copy Data

gaioladigital — Tue, 11 Feb 2025 18:47:08 GMT

I'm ingesting an oracle table using COPY DATA and turning it into a delta table in Fabric Lake house. However, the table I'm pulling takes an hour. I wanted to create a rule, where the next time I do a copy data, I go to this generated delta table to get the highest value in the sequential column. Then the next time I do the copy data, it uses the parameter, based on what has already been replicated. How can I do this?

Re: Implementing Incremental Ingestion in Fabric Lakehouse from an Oracle Table with Copy Data

Anonymous — Wed, 12 Feb 2025 02:33:16 GMT

Hi gaioladigital

Here are some of my thoughts on the question you asked:

After loading the data, use a SQL query to get the highest value of the order column from the Delta table.

SELECT MAX(sequential_column) AS max_value FROM DeltaTable;

Store the query result in a parameter for later use.

In the COPY DATA activity, add a filter condition to extract only the records where the sequential column value is greater than max_value. Add the filter condition in the query of the source dataset:

SELECT * FROM OracleTable WHERE sequential_column > @max_value;

Where @max_value is the parameter stored previously.

Configure Oracle database in a copy activity - Microsoft Fabric | Microsoft Learn

Append new data to the existing Delta table.

After each incremental load, use a SQL query to get the new highest value from the Delta table.

Run the pipeline with scheduled refresh to ensure data is incrementally loaded regularly.

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

topic Re: Implementing Incremental Ingestion in Fabric Lakehouse from an Oracle Table with Copy Data in Pipelines

Implementing Incremental Ingestion in Fabric Lakehouse from an Oracle Table with Copy Data

Re: Implementing Incremental Ingestion in Fabric Lakehouse from an Oracle Table with Copy Data