Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Reprocess data present in ADLS

Hi, We have a requirement fir a scenario to reprocess old data using data factory pipeline.Here are the details Storage in ADLSGEN2 Landing zone(where the data will be stored in the same format as...
pipelines
  • jwinchell40's avatar
    jwinchell40
    2 years ago

    Anonymous - How is your landing zone structured?  Are you using a hierarchy for storing when the file(s) were ingested in the landing zone (Ex:  Year -> Month -> Day..). 

     

    When you read from whichever file path is in the Landing Zone or all dynamic, you can access some Metadata as part of the process; including the Modified Date of the file.  You can then use a filter to get rid of any data that is before the seed date you passed in.

     

    df = spark.read.format('json').load(<path>).select("*","_metadata.file_modification_time)  Or

    df = spark.read.format('json').load(<path).select("*").filter("_metadata.file_modification_time" > "<date>")

jwinchell40's avatar
jwinchell40
Resolver III
2 years ago

Anonymous - How is your landing zone structured?  Are you using a hierarchy for storing when the file(s) were ingested in the landing zone (Ex:  Year -> Month -> Day..). 

 

When you read from whichever file path is in the Landing Zone or all dynamic, you can access some Metadata as part of the process; including the Modified Date of the file.  You can then use a filter to get rid of any data that is before the seed date you passed in.

 

df = spark.read.format('json').load(<path>).select("*","_metadata.file_modification_time)  Or

df = spark.read.format('json').load(<path).select("*").filter("_metadata.file_modification_time" > "<date>")