Power BI is turning 10! Tune in for a special live episode on July 24 with behind-the-scenes stories, product evolution highlights, and a sneak peek at what’s in store for the future.
Save the dateEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
I am trying to load JSON files from a Data Lake into SQL DW using Pipelines
I have a table LOG_PROCESSED_FILE and a stored Procedure. at the end of the process the SP records the processed File
And the current state is that I have processed 2 files and a new file is waiting to be loaded
1. GetJSONFiles Get Meta data activity File List 'Child Items'
Here is the output
{ "childItems": [ { "name": "A.json", "type": "File" }, { "name": "B.json", "type": "File" }, { "name": "C.json", "type": "File" } ], "effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (UK South)", "executionDuration": 1, "durationInQueue": { "integrationRuntimeQueue": 10 }, "billingReference": { "activityType": "PipelineActivity", "billableDuration": [ { "meterType": "AzureIR", "duration": 0.016666666666666666, "unit": "Hours" } ] } }
2. Lookup LookupProcessedFiles SELECT FileName FROM [framework].[PROCESSED_FILE_LOG] connected to my data warehouse
Here is the output
{ "count": 2, "value": [ { "FileName": "A.json" }, { "FileName": "B.json" } ], "effectiveIntegrationRuntime": "AutoResolveIntegrationRuntime (UK South)", "billingReference": { "activityType": "PipelineActivity", "billableDuration": [ { "meterType": "AzureIR", "duration": 0.016666666666666666, "unit": "Hours" } ], "totalBillableDuration": [ { "meterType": "AzureIR", "duration": 0.016666666666666666, "unit": "Hours" } ] }, "durationInQueue": { "integrationRuntimeQueue": 0 } }
3. A Filter. FilterProcessedFiles
Items: @activity('GetJsonFiles').output.childItems
Condition :
Solved! Go to Solution.
All sorted. The condition needed a slight change
@IF(empty(activity('LookupProcessedFiles').output.value), true, not(contains(string(activity('LookupProcessedFiles').output.value), item().name)))
Now wrapped in String
Hi, @DebbieE
Glad you found your problem, and you can mark your approach as a solution to help more people with similar problems find answers. Thanks for your understand.
Best Regards,
Yang
Community Support Team
All sorted. The condition needed a slight change
@IF(empty(activity('LookupProcessedFiles').output.value), true, not(contains(string(activity('LookupProcessedFiles').output.value), item().name)))
Now wrapped in String
This is your chance to engage directly with the engineering team behind Fabric and Power BI. Share your experiences and shape the future.
User | Count |
---|---|
5 | |
4 | |
2 | |
2 | |
2 |
User | Count |
---|---|
18 | |
15 | |
4 | |
4 | |
4 |