Forum Discussion

DebbieE's avatar
DebbieE
Community Champion
1 year ago
Solved

Pipeline. Delta Load JSON Files. Filtering out processed files not working

I am trying to load JSON files from a Data Lake into SQL DW using Pipelines   I have a table LOG_PROCESSED_FILE and a stored Procedure. at the end of the process the SP records the processed File ...
Data Pipeline
  • DebbieE's avatar
    DebbieE
    1 year ago

    All sorted. The condition needed a slight change

     

    IF(empty(activity('LookupProcessedFiles').output.value), true, not(contains(string(activity('LookupProcessedFiles').output.value), item().name)))

     

    Now wrapped in String

DebbieE's avatar
DebbieE
Community Champion
1 year ago

All sorted. The condition needed a slight change

 

IF(empty(activity('LookupProcessedFiles').output.value), true, not(contains(string(activity('LookupProcessedFiles').output.value), item().name)))

 

Now wrapped in String