Limitation of filter activity in foreach loop

[Anonymous]

I have met a limitation where I want to use a filter activity inside a for each loop and where I want to use an attribute from my For Loop item() as my filtering condition in the filter activity. However when setting the condition of the Filter activity item() refers to the input of the filter activity and not the item() of the for each activity.

I can work around this by adding item().object_name from the ForEach activity to a variable before passing it to the filter condition. But then I cannot run the for each loop in parallell which is a requirement in my case. Is there another way around this without adding nestled pipelines? It seems strange that both the filter activity and the for loop has the same item() syntax. Can the "ForEach item()" be accessed some other way?

https://learn.microsoft.com/en-us/azure/data-factory/control-flow-filter-activity

[pquispe]

News about this topic?

[Anonymous]

Hi @Anonymous 

Since you provided a link from azure, there may have been a misunderstanding.

As mentioned above, perhaps you can consider using variables.

Solved: Re: How to do nested for loops in data factory - Microsoft Fabric Community

Or try what Nico_Saar says.

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Anonymous]

In Azure data factory variables are global - meaning you cannot use them inside for loops running in parallell: https://learn.microsoft.com/en-us/azure/data-factory/control-flow-for-each-activity#limitations-and-...

Is Variables in Microsoft Fabric Data factory pipelines different than ADF? Because I need my ForEach to run in parallell... Documentation for Fabric data factory is a lot more limited than Azure data factory: https://learn.microsoft.com/en-us/fabric/data-factory/foreach-activity

[Nico_Saar]

In the Foreach container you are already at the level of an item. The filter activity is intended more for filtering out multiple inputs that meet certain criteria.
Would it make sense for your use case to place the filter activity outside the loop container and to run through the foreach case in parallel with the already filtered items?
Or, if not, use a switch or if activity instead of the filter activity, for example?

Regards

[Anonymous]

Thanks, I appreciate your answer! I can try to describe the workflow a bit more carefully.

What I’m trying to accomplish is to be able to move pipelines between workspaces (environments) without having to update anything codewise or metadata wise. We are using a metadata driven framework where we want to parameterize which notebook or pipeline to be run. Currently when parameterizing the notebook activity you need to specify the workspace and notebook / pipeline id and not the name of the item. This means that the metadata in Dev and Prod will be different since the item id is unique between workspaces. As a workaround for this I’m using the Fabric API to lookup fabric items in the current workspace of the pipeline: https://learn.microsoft.com/en-us/rest/api/fabric/core/items/list-items?tabs=HTTP 

Outside the ForEach I have one lookup “Get Job Objects” which lists which objects (notebooks or pipelines to run).

I also do an API lookup to the Fabric API. To get all items in the workspace (there is no way to query based on an item name in the API, you get everything for a specific item type).

I then would like to use the filter activity to use the name for the specific item I’m looping (item().object_name) to filter the item list from the API to get which ID to use for paramterization of the execute pipeline (or notebook) activity.

[xhead]

Thanks @Anonymous 

I am trying to do exactly what you are saying here, and hitting the same issue. I have an array of notebook names to call, and a ForEach loop that iterates and executes a Notebook task, but that task needs the ID of the notebook, not the name of the notebook. So I get a list of notebook names and their IDs in a different call. 

I have two questions:

1. Did you figure out a workaround to filter the collection of notebook metadata using the ForEach item? 

2. I've been using a Fabric notebook to retrieve the list of notebooks and their IDs rather than a Web call (to the REST API I presume). 

import sempy.fabric as fabric
import pandas as pd

items = [fabric.list_items(workspace=ws).query('Type=="Notebook"') for ws in fabric.list_workspaces().query('`Is On Dedicated Capacity` == True').Id]

jsItems = items[0].to_json(orient="records")

mssparkutils.notebook.exit(jsItems)

 I am betting that your Web call to the REST API is much faster but I was lazy - can you share your Web code?

[Anonymous]

1) No proper solution but you can work around it by instead of executing a notebook directly, invoke a child pipeline with two pipeline object parameters:

list_api_items -> input the whole web-call body: @activity('List items').output.value
for_loop_item -> input @item()

In the child pipeline you can then filter the list_api_items parameter as you would do normally.

However, we use a metadata driven orchestration framework. What we ended up doing is that we created a separate pipeline named "GetFabricItems". When we run that pipeline we retrive all items and their object id's for the current workspace and insert them into a database table. When we then retrive the lookup "Get Job Objects" (as shown in my screenshot in previous post) we will get the pipeline name and the pipeline id etc. Seems to be working well so far. We just need to make sure that we populate the fabric item table in our framework database before running any pipelines.

2) The API-call is:

For all items: https://api.fabric.microsoft.com/v1/workspaces/@{pipeline().DataFactory}/items

For specific type of items: https://api.fabric.microsoft.com/v1/workspaces/@{pipeline().DataFactory}/warehouses/ 
https://api.fabric.microsoft.com/v1/workspaces/@{pipeline().DataFactory}/notebooks/ 

With the specific api-calls you get som additional information based on the type. For example you get connection string for warehouses.

Documentation is here: https://learn.microsoft.com/en-us/rest/api/fabric/core/items/get-item?tabs=HTTP 

[Anonymous]

Hi @Anonymous 

This forum is for discussing issues related to the Fabric Data Factory, for Azure issues you can go to the following link, there will be more professional people to help you solve the problem:

Azure Data Factory - Microsoft Community Hub

Here are some possible ideas for your reference:

To solve this problem without using nested pipes, you can consider using the Set Variable activity in the ForEach loop to store the current item.

Inside the ForEach, add a Set Variable activity to store item().object_name.

Set the variable to @item().object_name.

Use the variable in the Filter activity’s condition, for example, @equals(variables('yourVariableName'), item().name).

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[Anonymous]

I'm using Data factory in Fabric. I think that was made clear from the screenshot?

As for the variable my intial post stated that using a variable - since it's not thread safe - and therfore does not work in parallell is not a viable option.