Forum Discussion
Dataflow Gen2 Fails to refresh - '_WriteToDataDestination' - 'Data source credentials are missing..
Hi dgeebs ,
Disabling “Enable staging” was a good tip, but in my case, the real issue came down to broken permissions and metadata bindings after deployment pipeline promotion.
- Reapplied Lakehouse permissions manually in the Prod workspace
→ Gave my user Build + Read/Write access again - Rebound the destination table in the Prod-stage Dataflow
→ Removed and re-added the Lakehouse output - Published and refreshed the dataflow worked successfully
After using pipelines, always verify Lakehouse permissions and rebind Dataflow destinations, even if everything looks fine.
Regards,
Akhil.- Reapplied Lakehouse permissions manually in the Prod workspace
Hi dgeebs ,
I faced the same issue where my Dataflow Gen2 would preview and save correctly, but fail on refresh with a "missing or invalid credentials" error at the WriteToDataDestination step.
Here’s what finally worked for me.
- Delete and Recreate the Lakehouse Connection
Go to Manage connections → Delete the old Lakehouse connection → Recreate it using OAuth (Organizational Account). - Reconfigure the Destination Table in Dataflow
Remove the current output table, and re-add it with auto-create table enabled or use a new table name. - Verify Lakehouse Permissions
Make sure your user has Build + Read/Write permissions on the Lakehouse (even if you're the owner). - Click "Publish" After Changes
After editing connections/destinations, don’t forget to publish the dataflow before refreshing. - Optional: Test with a Small Flow
Create a minimal test Dataflow writing to a new Lakehouse table to isolate the issue.
Regards,
Akhil.
Still no change. Weirdly enough, a dataflow from another workspace is able to write to the target table in the Lakehouse, but the Dataflow in the same workspace cannot.
For additional context, I tried setting up a deployment pipeline and the Dataflow Gen 2 and Lakehouse were both promoted through the Dev, Test, and Prod stages and what I'm finding is that the Dataflow from the Dev stage workspace can write to the table in the prod workspace, but the same dataflow in the prod stage workspace cannot write to the table in the prod workspace.
Is there a chance the Lakehouse permissions got messed up when the Lakehouse item was moved through the deployment pipeline? (I am Admin in all 3 workspaces that are a part of the pipeline)
