Forum Discussion

dzav's avatar
dzav
Icon for Advocate III rankAdvocate III
2 years ago
Solved

SQL endpoint sync issues

I've been experiencing 3 different issues related to the SQL endpoint not being refreshed quickly enough or at all when the undelying delta table is updated with new data. 1) A pipeline will fail wh...
pipelines
  • dzav's avatar
    dzav
    2 years ago

    frithjof_v , thank you! Looks like that was the optimal solution. I now how the following pattern:

    1) Load the bronze lakehouse

    2) A copy task to load the bronze warehouse from the bronze lakehouse

    3) A stored procedure to copy from the bronze warehouse to the silver warehouse

     

    I don't know how scalabale this is because we have over a hundred tables from Synapse Link and adding tables to a copy task is a bit onerous, but for now this seems to be working.

dzav's avatar
dzav
Icon for Advocate III rankAdvocate III
2 years ago

Thanks! Is anyone aware if Microsoft is treating this issue as a bug and something that they are working on fixing? Support has basically said that since the sql endpoint has to sync metadata, we shouldn't expect the delta to match. In other words, this is working as intended according to Microsoft support. Practically, from an end user perspective, this is super broken.

  • lbendlin's avatar
    lbendlin
    Icon for Super User rankSuper User
    2 years ago

    Voice your opinion with your Microsoft reps.  Raise tickets.  Vote for ideas.