Forum Discussion

pafnuty's avatar
pafnuty
Frequent Visitor
2 years ago
Solved

Filter in $project cursor in copy activity using Atlas MongoDB connector doesn't work

I have MongoDB Atlas as a source and then upload this data into Fabric DWH with some mappings
I've set up a $project cursor method in the "Source" tab of the Data Pipeline Copy Activity

It works fine when I write it like this:  {field1: 1, field2: 1, field3: 1}

But in my case, I need to add more complex logic with filtering for a particular field, like:

 

field1: 1, 
field2: 1,
results: {
$filter: {
input: '$results',
as: 'result',
cond: {
$in: [
'$$result.step',
[
'Value1',
'Value2',
'Value3',
'Value4'
]
]
}
}
}
}

But the fact that this query works with aggregate() function in MongoDB

 

What can be the issue using this projection within copy activity?

  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi pafnuty ,

     

    I have a suggestion that you can create an Azure Function to handle complex filtering logic. This Azure Function is then called in the data factory to process the data and pass the results to the next copy activity.

     

    Best Regards,
    Adamk Kong

     

    If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

3 Replies

    • pafnuty's avatar
      pafnuty
      Frequent Visitor

      Thanks Anonymous 
      Seems like this is not supported by Azure Data Factory

      Do you think there is some other way of applying this filtering with the help of Fabric?
      I have really a lot of data in case I'm not using filtering, which increase processing time in tens times

      • Anonymous's avatar
        Anonymous
        Not applicable

        Hi pafnuty ,

         

        I have a suggestion that you can create an Azure Function to handle complex filtering logic. This Azure Function is then called in the data factory to process the data and pass the results to the next copy activity.

         

        Best Regards,
        Adamk Kong

         

        If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.