Forum Discussion

AmanKalra's avatar
AmanKalra
Frequent Visitor
4 months ago
Solved

Dataflow Gen 2 is Failing with Scheduled refresh

Hi everyone,   I have a dataflow Gen 2 which is failing with Scheduled refresh and throwing below error but same is working via On-demand .   Its failing during WritetoDataDestination step to war...
admin
  • Olufemi7's avatar
    Olufemi7
    4 months ago

    Hello AmanKalra,
    Seen this before, on-demand works, scheduled fails at `WriteToDataDestination`. Usually points to the destination credentials not being valid when the scheduled run starts.
    Open the failed run and expand `WriteToDataDestination`. `ActionUserFailure` is too generic, the inner error is what matters.
    Then recheck the warehouse connection in the dataflow and re-enter credentials.
    Doc for this covers what gets validated during refresh for data destinations:
    Dataflow Gen2 data destinations validation rules 
    Append or Replace mode on the destination?

BarbaraC's avatar
BarbaraC
Icon for Advocate I rankAdvocate I
4 months ago

Hi,

the on-demand vs scheduled discrepancy is a classic symptom of a credentials/ownership issue with Warehouse as destination.

First thing to try: run it manually as the owner of the dataflow and see what happens:

  • If it works: the issue is with how the scheduled trigger authenticates. Check that the schedule is configured under the owner's account and that their credentials are not expired.
  • If it fails too: then it's a deeper credentials/connection issue. Try the following:
  1. Check who owns the dataflow: scheduled refreshes run under the owner's identity, not the user who set up the schedule. Go to workspace settings, confirm the owner, and make sure their Warehouse connection is authenticated and up to date.
  2. Re-authenticate the Warehouse connection. Make sure this is done by the owner identified in step 1. Go to Manage connections, delete the existing one and recreate it with OAuth/Organizational Account. Then remove and re-add the data destination in the dataflow, publish, and try again.
  3. Check the detailed refresh logs. The top-level error message doesn't give you much to work with. Try opening the refresh history, drilling into the failed run and looking at the per-step details: the real error is usually buried there (null type mismatch, 429 throttling, missing credentials on source, etc.).

Good luck, let us know what the logs say!

Best regards!

PS: If you find this post helpful consider leaving kudos or mark it as solution 😊