topic Re: Data Pipeline executes multiple times in Pipelines

Data Pipeline executes multiple times

Matthew_G — Wed, 22 Nov 2023 05:02:51 GMT

I have a pipleline and when it runs, either manually or scheduled, it executes multiple times. This seems to occur with any data pipeline we create. Is there something I am doing wrong?

To test I created a stored proc in a warehouse that inserts a date into a table.

create table dbo.runCheck (

runDate DATETIME2(0)

)

create procedure dbo.p_insertDate

insert into dbo.runCheck(runDate)

select getdate()

I then create a new data pipeline and add the stored proc activity.

Confrim table is empty

Execute the pipline and check monitoring hub, I can see it is running multiple times

Check the count again and I see there are multiple inserts

I'm not sure if I am doing something wrong or if there is a bug.

Re: Data Pipeline executes multiple times

Anonymous — Wed, 22 Nov 2023 10:38:11 GMT

Hi Matthew_G
Thanks for using Fabric Community.
Apologies for the issue you have been facing. Currently this is not a bug.
When I tried to repro the scenario, the pipeline ran only once.

This might require a deeper investigation from our engineering team about your workspace, the pipeline details and the logic behind it to properly understand what might be happening.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Please provide the ticket number here as we can keep an eye on it.

Thank you.

Re: Data Pipeline executes multiple times

Matthew_G — Wed, 22 Nov 2023 10:46:10 GMT

Thank you! I have logged ticket with TrackingID#2311170050002237 . I thought it could be related to our enviroment and our ops team is also investigating.

Re: Data Pipeline executes multiple times

Anonymous — Wed, 22 Nov 2023 10:54:39 GMT

Hi Matthew_G
Thanks for the details. Please continue using Fabric Community for any help regarding your queries.

Re: Data Pipeline executes multiple times

Matthew_G — Wed, 22 Nov 2023 11:28:58 GMT

We found the issue was a firewall related. There were retries because *.analysis.windows.net was not allowed. This caused multiple instances to be fired off when we manually tried to run the pipeline. Following these instructions worked: https://learn.microsoft.com/en-us/power-bi/enterprise/power-bi-allow-list-urls

Re: Data Pipeline executes multiple times

Anonymous — Wed, 22 Nov 2023 11:36:41 GMT

Thanks for the details Matthew_G .