Forum Discussion

KarolinaKa's avatar
KarolinaKa
Frequent Visitor
1 year ago
Solved

Views do not refresh

Hello!   I'm currently trying to solve the mystery why my scheduled refresh for tables at night do not refresh properly sometimes even though when I run manually that update pipeline, it always wor...
  • KarolinaKa's avatar
    KarolinaKa
    1 year ago

    Hey,

     

    it is still a misery to me, because:

    a) I turned off other scheduled refreshes and left just this one

    b) if I run the pipeline manually, everything works just fine (data lies to DWH)

    c) if I run the pipeline with scheduled refresh during the day, everything works just fine

    d) if I come in the morning and check nightly scheduled refresh it always shows as 'succeeded' but raw tables in the DWH are not updated

    e) My update pipeline architecture: Copy data activity (Mysql to LH data from the last 2 days based on updated_at field) -> Wait activity (180s) -> Upsert BULK script activity (which checks LH and DWH data based on id and retrieves LH new data to the DWH).  Quite simple architecture and works for the other created DWH (quite similar to this one by tables, size structures) while this one lacks to update the tables in the DWH.

    f) In the morning I deactivate copy and wait activities in this pipeline and only run manually update script again and everything works just fine (new data lies in the DWH)

    g) Crazy, huh?