Error using Dataflow activity in Pipeline with dynamic content

[AnibalBenedetto]

Hello!, seems to be an issue with pipelines. When you add a Dataflow “activity” and set it with dynamic content (introducing his Dataflow object ID) you can Validate, Save and Run correctly, but if you close it, you won’t be able to open again.

I am trying to build a pipeline to run a list of Dataflows in a ForEech Activity.

Thanks!

[DiegoPLAB]

Hi everybody,

I have the same issue on a pipeline that invokes a dataflow with dynamic content.

However, in my case the dataflow is inside a ForEach iterator and I was trying to invoke a dataflow by its name building an expression like this:

@concat('Clean_', item())

The result of the expression is correct: dataflows with the expected names exist in my workspace. However, when I ran the pipeline I got an error saying that dataflows doesn't exist.

Now I can't open anymore the pipeline.

It would be nice if you can dynamically invoke a dataflow using its name rather than the internal ID.

Thanks.

[v-cboorla-msft]

Hi @AnibalBenedetto 

Thanks for using Microsoft Fabric Community.

I tried to repro the scenario and got the same error when giving the "Dataflow Object ID" dynamically.

Instead of hard coding the value we can give the input as a parameter as shown in the screenshots below:

You can open the pipeline now even after closing it without any error.

Hope this helps. Please do let us know if you any further queries.

[AndyDDC]

Hi @AnibalBenedetto are you having issues opening the pipeline itself after you closed it?  I tried explicitly stating the dataflow id and also using a variable, and both seemed to validate and run fine. I could close the pipeline and re-open.

Have you tried using a variable and seeing if that makes a difference?

Also, what does the FSON View look like? Click View > View JSON Code (after you've got the pipeline working again) just to see what the JSON looks like

[AnibalBenedetto]

Thanks for your time!
Yes, I can’t open the pipeline artifact after close it.
I don’t know way, but I can’t reproduce your recommendations using the ID value or a Variable.
The JSON Code is empty in my case and I can’t edit / recover the original pipeline.

[AndyDDC]

Can you try recreating the pipeline and using the variable?

[AnibalBenedetto]

Yes, but I can't open it again in any case:

The JSON is empty:

Thanks!

[AndyDDC]

Hmm you may have found a bug! 

@v-nikhilan-msft one to log with the MS team?

[v-nikhilan-msft]

Hi @AndyDDC , @AnibalBenedetto ,

I tried to repro the scenario and got the same error when giving the "Dataflow Object ID" dynamically.

Instead of hard coding the value we can give the input as a parameter as shown in the screenshots below:

You can open the pipeline now even after closing it without any error.

Can you please try this @AnibalBenedetto  and confirm, if this works?
Thanks.

[AnibalBenedetto]

Hello!, I want a dynamic value, so Parameter is not useful in this case. In any case, I get the same error.

[v-nikhilan-msft]

Thanks for the update @AnibalBenedetto .
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.
Thanks 

[ajarora]

One useful thing to gather here for a repro will be the json of the corrupted pipeline. And then using that for repro.

[AnibalBenedetto]

Hello, I had created a new Dataflow and then I created a Pipeline that calls to that Dataflow using its dataflowId.

Here you have the json code from the pipeline (before closing it):

{
    "name": "toDelete",
    "objectId": "d7943d13-dfb9-4ac8-8423-0b00662751a7",
    "properties": {
        "activities": [
            {
                "name": "Dataflow1",
                "type": "RefreshDataflow",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "typeProperties": {
                    "dataflowId": {
                        "value": "712a0555-226f-49eb-b298-75038b965749",
                        "type": "Expression"
                    },
                    "workspaceId": "740736a8-b130-4ae4-bbef-819062cf67c5",
                    "notifyOption": "NoNotification"
                }
            }
        ],
        "annotations": [],
        "lastModifiedByObjectId": "a1844513-17a8-4328-aae5-9d80baf0a8dc",
        "lastPublishTime": "2023-11-16T17:15:30Z"
    }
}

Thanks!

[v-cboorla-msft]

Hi @AnibalBenedetto 

Apologies for the delay in response.
Follow up to see if you have a resolution yet on your issue.
In case if you have any resolution please do share that same with the community as it can be helpful to others.

This might require a deeper investigation from our engineering team about your workspace, the pipeline details and the logic behind it to properly understand what might be happening. 
In case if you didn't get any resolution.

Please go ahead and raise a support ticket to reach our support team: https://support.fabric.microsoft.com/support

Please provide the ticket number here as we can keep an eye on it.

Thank you.

[AndyDDC]

Yes but I didn't get an error when either using hardcoded dataflow id or the variable