Forum Discussion

DebbieE's avatar
DebbieE
Community Champion
1 year ago
Solved

Pipeline. Delta Load JSON Files. Filtering out processed files not working

I am trying to load JSON files from a Data Lake into SQL DW using Pipelines   I have a table LOG_PROCESSED_FILE and a stored Procedure. at the end of the process the SP records the processed File ...
Data Pipeline
  • DebbieE's avatar
    DebbieE
    1 year ago

    All sorted. The condition needed a slight change

     

    IF(empty(activity('LookupProcessedFiles').output.value), true, not(contains(string(activity('LookupProcessedFiles').output.value), item().name)))

     

    Now wrapped in String

Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi, DebbieE 

Glad you found your problem, and you can mark your approach as a solution to help more people with similar problems find answers. Thanks for your understand.

Best Regards,
Yang
Community Support Team