Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Don't miss out! 2025 Microsoft Fabric Community Conference, March 31 - April 2, Las Vegas, Nevada. Use code MSCUST for a $150 discount. Prices go up February 11th. Register now.

Reply
pafnuty
Frequent Visitor

Filter in $project cursor in copy activity using Atlas MongoDB connector doesn't work

I have MongoDB Atlas as a source and then upload this data into Fabric DWH with some mappings
I've set up a $project cursor method in the "Source" tab of the Data Pipeline Copy Activity

It works fine when I write it like this:  {field1: 1, field2: 1, field3: 1}

But in my case, I need to add more complex logic with filtering for a particular field, like:

 

field1: 1, 
field2: 1,
results: {
$filter: {
input: '$results',
as: 'result',
cond: {
$in: [
'$$result.step',
[
'Value1',
'Value2',
'Value3',
'Value4'
]
]
}
}
}
}

But the fact that this query works with aggregate() function in MongoDB

 

What can be the issue using this projection within copy activity?

1 ACCEPTED SOLUTION

Hi @pafnuty ,

 

I have a suggestion that you can create an Azure Function to handle complex filtering logic. This Azure Function is then called in the data factory to process the data and pass the results to the next copy activity.

 

Best Regards,
Adamk Kong

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

View solution in original post

3 REPLIES 3
v-kongfanf-msft
Community Support
Community Support

Hi @pafnuty ,

 

Ensure that the MongoDB Atlas connector configured in the replication activity supports the use of $filter in the $project phase.

You can refer to the following official documentation:

Copy data from or to MongoDB Atlas - Azure Data Factory & Azure Synapse | Microsoft Learn

 

Best Regards,
Adamk Kong

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Thanks @v-kongfanf-msft 
Seems like this is not supported by Azure Data Factory

Do you think there is some other way of applying this filtering with the help of Fabric?
I have really a lot of data in case I'm not using filtering, which increase processing time in tens times

Hi @pafnuty ,

 

I have a suggestion that you can create an Azure Function to handle complex filtering logic. This Azure Function is then called in the data factory to process the data and pass the results to the next copy activity.

 

Best Regards,
Adamk Kong

 

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Helpful resources

Announcements
Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Prices go up Feb. 11th.

JanFabricDE_carousel

Fabric Monthly Update - January 2025

Explore the power of Python Notebooks in Fabric!

JanFabricDW_carousel

Fabric Monthly Update - January 2025

Unlock the latest Fabric Data Warehouse upgrades!

JanFabricDF_carousel

Fabric Monthly Update - January 2025

Take your data replication to the next level with Fabric's latest updates!