Forum Discussion
Dataflow Gen2 Fails to refresh - '_WriteToDataDestination' - 'Data source credentials are missing..
Hi dgeebs ,
Disabling “Enable staging” was a good tip, but in my case, the real issue came down to broken permissions and metadata bindings after deployment pipeline promotion.
- Reapplied Lakehouse permissions manually in the Prod workspace
→ Gave my user Build + Read/Write access again - Rebound the destination table in the Prod-stage Dataflow
→ Removed and re-added the Lakehouse output - Published and refreshed the dataflow worked successfully
After using pipelines, always verify Lakehouse permissions and rebind Dataflow destinations, even if everything looks fine.
Regards,
Akhil.- Reapplied Lakehouse permissions manually in the Prod workspace
Hi dgeebs ,
Disabling “Enable staging” was a good tip, but in my case, the real issue came down to broken permissions and metadata bindings after deployment pipeline promotion.
- Reapplied Lakehouse permissions manually in the Prod workspace
→ Gave my user Build + Read/Write access again - Rebound the destination table in the Prod-stage Dataflow
→ Removed and re-added the Lakehouse output - Published and refreshed the dataflow worked successfully
After using pipelines, always verify Lakehouse permissions and rebind Dataflow destinations, even if everything looks fine.
Regards,
Akhil.
Thanks Anonymous for your help and support! Apologies for disappearing for a while, I was finally able to determine what the issue was with my dataflow.
In my case, I had a dataflow which used a function to fetch data via a REST API for each row in a source table. Although the preview was succeeding in the PowerQuery UI, after further digging I found that one of the data source API's had become inaccessible due to broken credentials. Once I removed the row from the source table that was failing to collect it's data from it's respective data source API, the dataflow succeeded.
What I or fellow readers can take from this is that even if the below error message shows up on the step XXX_WriteToDataDestination:
There was a problem refreshing the dataflow: 'Data source credentials are missing or invalid. Please update the connection credentials in settings, and try again.'.
It may not indicate that the error lies in specifically writing your data to your destination lakehouse - it may be in regards to credential issues with fetching data from an external data source somewhere along the way.
Make sure to check every step of your dataflow for errors and you'll likely find where your connection issues lie.