topic Re: Error using Dataflow activity in Pipeline with dynamic content in Data Engineering

Error using Dataflow activity in Pipeline with dynamic content

AnibalBenedetto — Fri, 10 Nov 2023 11:03:51 GMT

Hello!, seems to be an issue with pipelines. When you add a Dataflow “activity” and set it with dynamic content (introducing his Dataflow object ID) you can Validate, Save and Run correctly, but if you close it, you won’t be able to open again.

I am trying to build a pipeline to run a list of Dataflows in a ForEech Activity.

Thanks!

Re: Error using Dataflow activity in Pipeline with dynamic content

AndyDDC — Fri, 10 Nov 2023 12:33:31 GMT

Hi @AnibalBenedetto are you having issues opening the pipeline itself after you closed it? I tried explicitly stating the dataflow id and also using a variable, and both seemed to validate and run fine. I could close the pipeline and re-open.

Have you tried using a variable and seeing if that makes a difference?

Also, what does the FSON View look like? Click View > View JSON Code (after you've got the pipeline working again) just to see what the JSON looks like

Re: Error using Dataflow activity in Pipeline with dynamic content

v-cboorla-msft — Fri, 10 Nov 2023 12:40:00 GMT

Hi @AnibalBenedetto

Thanks for using Microsoft Fabric Community.

I tried to repro the scenario and got the same error when giving the "Dataflow Object ID" dynamically.

Instead of hard coding the value we can give the input as a parameter as shown in the screenshots below:

You can open the pipeline now even after closing it without any error.

Hope this helps. Please do let us know if you any further queries.

Re: Error using Dataflow activity in Pipeline with dynamic content

AnibalBenedetto — Fri, 10 Nov 2023 13:33:57 GMT

Thanks for your time!
Yes, I can’t open the pipeline artifact after close it.
I don’t know way, but I can’t reproduce your recommendations using the ID value or a Variable.
The JSON Code is empty in my case and I can’t edit / recover the original pipeline.

Re: Error using Dataflow activity in Pipeline with dynamic content

AndyDDC — Fri, 10 Nov 2023 18:12:33 GMT

Can you try recreating the pipeline and using the variable?

Re: Error using Dataflow activity in Pipeline with dynamic content

AnibalBenedetto — Fri, 10 Nov 2023 19:03:38 GMT

Yes, but I can't open it again in any case:

The JSON is empty:

Thanks!

Re: Error using Dataflow activity in Pipeline with dynamic content

AndyDDC — Fri, 10 Nov 2023 19:45:06 GMT

Hmm you may have found a bug!

@Anonymous one to log with the MS team?

Re: Error using Dataflow activity in Pipeline with dynamic content

Anonymous — Fri, 10 Nov 2023 19:48:49 GMT

Hi @AndyDDC , @AnibalBenedetto ,

I tried to repro the scenario and got the same error when giving the "Dataflow Object ID" dynamically.

Instead of hard coding the value we can give the input as a parameter as shown in the screenshots below:

You can open the pipeline now even after closing it without any error.

Can you please try this @AnibalBenedetto and confirm, if this works?
Thanks.

Re: Error using Dataflow activity in Pipeline with dynamic content

AndyDDC — Fri, 10 Nov 2023 19:56:56 GMT

Yes but I didn't get an error when either using hardcoded dataflow id or the variable

Re: Error using Dataflow activity in Pipeline with dynamic content

AnibalBenedetto — Fri, 10 Nov 2023 20:04:34 GMT

Hello!, I want a dynamic value, so Parameter is not useful in this case. In any case, I get the same error.

Re: Error using Dataflow activity in Pipeline with dynamic content

Anonymous — Fri, 10 Nov 2023 20:32:17 GMT

Thanks for the update @AnibalBenedetto .
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.
Thanks

Re: Error using Dataflow activity in Pipeline with dynamic content

DiegoPLAB — Wed, 15 Nov 2023 09:48:08 GMT

Hi everybody,

I have the same issue on a pipeline that invokes a dataflow with dynamic content.

However, in my case the dataflow is inside a ForEach iterator and I was trying to invoke a dataflow by its name building an expression like this:

@concat('Clean_', item())

The result of the expression is correct: dataflows with the expected names exist in my workspace. However, when I ran the pipeline I got an error saying that dataflows doesn't exist.

Now I can't open anymore the pipeline.

It would be nice if you can dynamically invoke a dataflow using its name rather than the internal ID.

Thanks.

Re: Error using Dataflow activity in Pipeline with dynamic content

ajarora — Wed, 15 Nov 2023 20:13:33 GMT

One useful thing to gather here for a repro will be the json of the corrupted pipeline. And then using that for repro.

Re: Error using Dataflow activity in Pipeline with dynamic content

AnibalBenedetto — Thu, 16 Nov 2023 17:32:42 GMT

Hello, I had created a new Dataflow and then I created a Pipeline that calls to that Dataflow using its dataflowId.

Here you have the json code from the pipeline (before closing it):

{ "name": "toDelete", "objectId": "d7943d13-dfb9-4ac8-8423-0b00662751a7", "properties": { "activities": [ { "name": "Dataflow1", "type": "RefreshDataflow", "dependsOn": [], "policy": { "timeout": "0.12:00:00", "retry": 0, "retryIntervalInSeconds": 30, "secureOutput": false, "secureInput": false }, "typeProperties": { "dataflowId": { "value": "712a0555-226f-49eb-b298-75038b965749", "type": "Expression" }, "workspaceId": "740736a8-b130-4ae4-bbef-819062cf67c5", "notifyOption": "NoNotification" } } ], "annotations": [], "lastModifiedByObjectId": "a1844513-17a8-4328-aae5-9d80baf0a8dc", "lastPublishTime": "2023-11-16T17:15:30Z" } }

Thanks!

Re: Error using Dataflow activity in Pipeline with dynamic content

v-cboorla-msft — Mon, 27 Nov 2023 04:07:43 GMT

Hi @AnibalBenedetto

Apologies for the delay in response.
Follow up to see if you have a resolution yet on your issue.
In case if you have any resolution please do share that same with the community as it can be helpful to others.

This might require a deeper investigation from our engineering team about your workspace, the pipeline details and the logic behind it to properly understand what might be happening.
In case if you didn't get any resolution.

Please go ahead and raise a support ticket to reach our support team: https://support.fabric.microsoft.com/support

Please provide the ticket number here as we can keep an eye on it.

Thank you.