Add new and last modified records to a delta table in Lakehouse Fabric from parquet files

[aliebrahimi64]

I have some parquet files related to GeneralLedger records in a Lakehouse and I want using pipeline to add only new and last modified records to a Delta table. What should I do?
I used combination of lookup activity and copy data, but doesn't work for me.
Any idea?

[HimanshuS-msft]

Hello @aliebrahimi64 

I am not sure if you have explored the option of dataflow . I will try to get the max date from delta table and the use this data as a filter on the all the records for from the paraquet files ( this means that all the records after that date is either modified or inserted ) . I can create a query in Dataflow to read the all the records from the delta table , so now we have to queries and now I will merge these two query to get that the new dataset . 

Let me know how it goes . If can share the some more info on the schema i can try to see if I can implement that on my side .
Thanks
HImanshu

[Anonymous]

Hi @aliebrahimi64 ,

Thanks for using Fabric Community.

As I understand you would like to append new data (present in parquet file) to existing delta table.
I guess you can acheive a solution if you use pyspark and notebooks.

I suggest to see this video so you might get some insights on how to move forward with this scenario.

AzDatabricks # 26:- How to Append data in delta table using merge function (youtube.com)

Hope it is helpful. Please let me know incase of further queries.

[Anonymous]

Hi @aliebrahimi64 ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .

[Anonymous]

Hi @aliebrahimi64 ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered. Otherwise, will respond back with the more details and we will try to help .