Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
AnibalBenedetto
Frequent Visitor

Error using Dataflow activity in Pipeline with dynamic content

Hello!, seems to be an issue with pipelines. When you add a Dataflow “activity” and set it with dynamic content (introducing his Dataflow object ID) you can Validate, Save and Run correctly, but if you close it, you won’t be able to open again.

 

AnibalBenedetto_0-1699613642381.png

 

 

AnibalBenedetto_1-1699613721498.png

I am trying to build a pipeline to run a list of Dataflows in a ForEech Activity.

Thanks!

 

14 REPLIES 14
DiegoPLAB
Advocate I
Advocate I

Hi everybody,

I have the same issue on a pipeline that invokes a dataflow with dynamic content.

However, in my case the dataflow is inside a ForEach iterator and I was trying to invoke a dataflow by its name building an expression like this:

@concat('Clean_', item())

The result of the expression is correct: dataflows with the expected names exist in my workspace. However, when I ran the pipeline I got an error saying that dataflows doesn't exist.

Now I can't open anymore the pipeline.

 

It would be nice if you can dynamically invoke a dataflow using its name rather than the internal ID.

 

Thanks.

v-cboorla-msft
Community Support
Community Support

Hi @AnibalBenedetto 

 

Thanks for using Microsoft Fabric Community.

I tried to repro the scenario and got the same error when giving the "Dataflow Object ID" dynamically.

vcboorlamsft_2-1699619500737.png

Instead of hard coding the value we can give the input as a parameter as shown in the screenshots below:

vcboorlamsft_3-1699619779208.pngvcboorlamsft_4-1699619865802.png

You can open the pipeline now even after closing it without any error.

 

Hope this helps. Please do let us know if you any further queries.

 

 

 

AndyDDC
Solution Sage
Solution Sage

Hi @AnibalBenedetto are you having issues opening the pipeline itself after you closed it?  I tried explicitly stating the dataflow id and also using a variable, and both seemed to validate and run fine. I could close the pipeline and re-open.

 

Have you tried using a variable and seeing if that makes a difference?

 

Also, what does the FSON View look like? Click View > View JSON Code (after you've got the pipeline working again) just to see what the JSON looks like

 

AndyDDC_0-1699619510058.png

 

 

AndyDDC_1-1699619524205.png

 

AndyDDC_2-1699619594546.png

 

Thanks for your time!
Yes, I can’t open the pipeline artifact after close it.
I don’t know way, but I can’t reproduce your recommendations using the ID value or a Variable.
The JSON Code is empty in my case and I can’t edit / recover the original pipeline.

Can you try recreating the pipeline and using the variable?

Yes, but I can't open it again in any case:

AnibalBenedetto_0-1699642635004.png

 

AnibalBenedetto_2-1699642799131.png

 

The JSON is empty:

AnibalBenedetto_3-1699642900351.png

 

Thanks!

 

 

 

Hmm you may have found a bug! 

@v-nikhilan-msft one to log with the MS team?

 

Hi @AndyDDC , @AnibalBenedetto ,

I tried to repro the scenario and got the same error when giving the "Dataflow Object ID" dynamically.

 

Instead of hard coding the value we can give the input as a parameter as shown in the screenshots below:

vnikhilanmsft_1-1699645686122.png

 

vnikhilanmsft_2-1699645686124.png

 

You can open the pipeline now even after closing it without any error.

Can you please try this @AnibalBenedetto  and confirm, if this works?
Thanks.

Hello!, I want a dynamic value, so Parameter is not useful in this case. In any case, I get the same error.

 

AnibalBenedetto_0-1699646409668.png

 

AnibalBenedetto_2-1699646471061.png

 

 

AnibalBenedetto_3-1699646521799.png

 

 

Thanks for the update @AnibalBenedetto .
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.
Thanks 

One useful thing to gather here for a repro will be the json of the corrupted pipeline. And then using that for repro.

Hello, I had created a new Dataflow and then I created a Pipeline that calls to that Dataflow using its dataflowId.

 

Here you have the json code from the pipeline (before closing it):

{
    "name": "toDelete",
    "objectId": "d7943d13-dfb9-4ac8-8423-0b00662751a7",
    "properties": {
        "activities": [
            {
                "name": "Dataflow1",
                "type": "RefreshDataflow",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "typeProperties": {
                    "dataflowId": {
                        "value": "712a0555-226f-49eb-b298-75038b965749",
                        "type": "Expression"
                    },
                    "workspaceId": "740736a8-b130-4ae4-bbef-819062cf67c5",
                    "notifyOption": "NoNotification"
                }
            }
        ],
        "annotations": [],
        "lastModifiedByObjectId": "a1844513-17a8-4328-aae5-9d80baf0a8dc",
        "lastPublishTime": "2023-11-16T17:15:30Z"
    }
}

 

Thanks!

Hi @AnibalBenedetto 

 

Apologies for the delay in response.
Follow up to see if you have a resolution yet on your issue.
In case if you have any resolution please do share that same with the community as it can be helpful to others.

This might require a deeper investigation from our engineering team about your workspace, the pipeline details and the logic behind it to properly understand what might be happening. 
In case if you didn't get any resolution.

Please go ahead and raise a support ticket to reach our support team: https://support.fabric.microsoft.com/support

 

Please provide the ticket number here as we can keep an eye on it.

 

Thank you.

Yes but I didn't get an error when either using hardcoded dataflow id or the variable 

Helpful resources

Announcements
FabricCarousel_June2024

Fabric Monthly Update - June 2024

Check out the June 2024 Fabric update to learn about new features.