Scheduled flow fails randomly with “Challenge Error” after changing credentials

[CreativeEnergy]

I’m having an issue with a flow based on Business Central. It used to run fine, but after changing the credentials to a service account, it started failing intermittently.

When I trigger the flow manually, it runs successfully every time. In the designer view, everything looks fine as well. However, when the flow runs on schedule, it sometimes works and sometimes fails, very inconsistently.

From the run history, the error I get is:

Challenge Error, Error Details: Data source credentials are missing or invalid. Please update the connection credentials in settings, and try again.

I already deleted and recreated all the connections, but that didn’t solve the issue.
Has anyone experienced this behavior before, or does anyone know what might cause scheduled runs to fail randomly like this?

[CreativeEnergy · Accepted solution]

I managed to get in contact with Microsoft support. At first they we're not able to point out what the reason for our issue was, but at some point we noticed that newly scheduled dataflows actually did run without issue. We came up with the idea to recreate each schedule and remove the old ones. This solved our issue and our dataflows have been running without issue for multiple hours now.

[tayloramy]

Hi @CreativeEnergy, 

Is it always the same error that you get? 

As far as I know, once you have connections set up properly, there should be no difference between refreshing the dataflow on demand, or on a schedule. 

Are you able to create a quick notebook that runs this dataflow as an on demand run via the Job Scheduler API? 
Job Scheduler - Run On Demand Item Job - REST API (Core) | Microsoft Learn

I think the end result here is going to be a ticket to Microsoft, but I'm curious if a scheduled "on demand run" via the API will work. 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[CreativeEnergy]

I'm not very familiar with this, so I'm afraid (without putting a lot of effort in it) I'm not able to do this. Where can I submit a ticket to Microsoft? As far as I know you need some extra license to get support, is that true?

[CreativeEnergy]

Thank you for your quick response. It might not be clear from my description, but the flow I'm referring to is a dataflow (Gen2) in Fabric. Do I still have to go into Power Automate?

[BeaBF]

@CreativeEnergy Have you tried re-authenticating? Sometimes this issue happens because the stored credentials or access token have expired. Try re-authenticating the connection in Fabric, do you know how to do it?

BBF

💡 Did I answer your question? Mark my post as a solution!

👍 Kudos are appreciated

🔥 Proud to be a Super User!

[CreativeEnergy]

I deleted the connection and re-created that, as explained in my initial post. Is that different then re-authenticating?

[BeaBF]

@CreativeEnergy Yes, it's a little different. 

Try to re-authenticate in this way:

1 Go to your workspace, open the Dataflow Gen2, then navigate to Settings >> Connections

2 Find the Business Central connection, click Edit connection >> choose Sign in >> enter the service account credentials again

3 Make sure you see the message 'Connection validated successfully'

4 Check that the service account has at least Contributor or Owner access to the workspace

5 Run a manual refresh to confirm it works, then let the next scheduled refresh run to verify

Let me know

BBF

[Natalie_iTalent]

Hello @CreativeEnergy,

The response you responded to has been removed due to inaccuracy and AI generated.

Best,

Natalie H.

Community Manager 

[tayloramy]

Hi @CreativeEnergy, 

While I don't have experience sepcifically with the BC connector, I do have experience troubleshooting Fabric. 

You say this works when you run it manually, but then fails when it is run on a schedule?  

That makes me think something wonky is happening where it is running under a different crednetial context when you run it manually. 

Can you share screenshots of the dataflow connection settings? 

Can you also share a screenshot of the refresh history showing it failing on a scheduled refresh but working on a manual one? 

If you found this helpful, consider giving some Kudos. If I answered your question or solved your problem, mark this post as the solution.

[CreativeEnergy]

Thank you for trying to helping me out!

While looking at multiple run histories, I noticed that some flows actually seem to work (sometimes) while running on schedule, so it seems it is not completely as described in my initial post. I'm sorry for that. Below you find the requested screenshots (if they are still of any use):

[BeaBF]

@CreativeEnergy I’ve just read this response, the issue could be related to timing or token conflicts, especially if multiple dataflows are sharing the same Business Central connection. When two or more refreshes start around the same time, Fabric might refresh the authentication token, which can cause one of them to fail with the “Challenge Error.”
My advice is to try to create a dedicated Business Central connection for each dataflow, even if they all use the same service account.

BBF

[CreativeEnergy]

I just implemented this. I have scheduled some runs. Let's hope this will solve my issue. I'll keep you posted!

[CreativeEnergy]

Unfortunately, this didn't do the trick. I created a dedicated connection for a flow, but again it resulted in the same error when with scheduled runs (while demand just worked).

Any more options to look into?