Forum Discussion

eurenergy's avatar
eurenergy
Frequent Visitor
1 year ago
Solved

Best practice for moving data from files to delta table in Fabric

Hi all, I set up a weekly scheduled data pipeline which copies data from Azure Datalake Storage Gen2 (ADLS gen2) to the Fabric LakeHouse.  It works perfectly, it simply takes all files added last w...
  • spencer_sa's avatar
    1 year ago

    We do this in three ways;
    1) if we have local copies of the files, we have a folder structure that has an unprocessed folder and a processed folder.  A pipeline with a Get Metadata activity on the unprocessed folder feeding a For Each activity.  Inside that is a Copy Data activity and some file copy/delete steps.  (No Move File activity 😞 )
    2) if all we have is a shortcut and we don't want to copy the files, we maintain a list of processed files in a table (using a notebook).  In a second notebook, we then do a left anti join on the files in the shortcut against the processed list and output the list of unprocessed files.  This then feeds a For Each activity to do the Copy Data and then using the first notebook append the processed file.
    3) If we use a shortcut *and* keep a local copy of the processed files, then we can do somethin like 2), just subbing in a directory listing instead of the processed file table.

    If this helps, please consider Accepting as a solution to help other people find it more easily.