Forum Discussion
Fabric Pipeline. Data flow. Removing items from a Filter with GetMetaData and Lookup
- 2 years ago
Perhaps you can do similar like below:
I have a Bronze Lakehouse and a Silver Lakehouse.
The files in my Bronze Lakehouse are as follows:
The files in my Silver Lakehouse are as follows:
I made a pipeline like this:
The Get Metadata activities get the Child items metadata from the File folder in Bronze lakehouse and Silver lakehouse, respectively.
The Filter activity removes the lookup.csv file from the output of the metadata activity from Bronze lakehouse:
Items: @activity('Get Metadata Bronze').output.childItems
Condition: @not(equals(item().name, 'lookup.csv'))
The Items in the ForEach activity is the output from the Filter activity:
Items: @activity('Filter Away Lookup file').output.ValueThe If Condition inside the ForEach activity:
Expression: @contains(activity('Get Metadata Silver').output.childItems, item())
The Copy activity if the If Condition is False:
After I run the pipeline, the Proje.csv file has been copied to Silver:
I don't know if Fabric Data Pipeline has any limits (like output size, number of items in collection, number of items in foreach activity, etc.) which needs to be taken into consideration or it can result in pipeline failure or unexpected results if the number of files in any of the folders grow above the limits.
HI DebbieE,
I think you need a template list or query result that used to compare with current items, or you can't define which not exist and use to filter.
Regards,
Xiaoxin Sheng
I would need some specific information to work with here for how I would go about that. This is all in a fabric pipeline
- Anonymous2 years agoNot applicable
Hi DebbieE,
Here is the document link about use dataflow in data pipeline, you can use M query editor to operation with query table records:
Use a dataflow in a pipeline - Microsoft Fabric | Microsoft Learn
Regards,
Xiaoxin Sheng