topic copy files from blob to lakehouse file in Pipelines

copy files from blob to lakehouse file

Nikhil234 — Fri, 19 Jul 2024 08:52:25 GMT

Hi,

I have a file that goes into blob everyday with a timestamp. I want to use a pipeline to dynamically get the latest file each day and load it to the files section in the lakehouse. The aim is to get the latest data. overwirte the file in the lakehosue and then use a notbook tp transform the data and load to a single table.

I have done all the other steps but i cant figure out how to get the correct file in the piepline.

I read the documentation on incrementally loading files based on modified date in the copy data tool but this functionality dont seem to be in the fabric pipeline.

How can i do this?

Re: copy files from blob to lakehouse file

NandanHegde — Fri, 19 Jul 2024 09:39:09 GMT

The below blog provides details to get the latest file details :

https://datasharkx.wordpress.com/2023/01/05/get-latest-folder-file-detail-from-azure-data-lake-storage-via-synapse-azure-data-factory-pipeline/

ADF is similar to data pipelines

Re: copy files from blob to lakehouse file

Anonymous — Mon, 22 Jul 2024 08:42:36 GMT

Hi Nikhil234 ,

Did NandanHegde reply solve your problem? If so, please mark it as the correct solution, and point out if the problem persists.

Best Regards,
Adamk Kong

Re: copy files from blob to lakehouse file

Nikhil234 — Mon, 22 Jul 2024 08:52:57 GMT

Hi NandanHegde. The blog was defintily helpful and i used elements of this in my soluation. Definitly recomend people read it. In the end i got the i used the get meta data - child items of the blob filtering on the modified date for within 3hrs of the pipline run time. i set the candence to allighn with the ingestion time of the daility file meaning i could pick up the file i needed each day. From there i simply put the output in the copy data activity and its been working great