Forum Discussion

dzav's avatar
dzav
Icon for Advocate III rankAdvocate III
2 years ago
Solved

SQL endpoint sync issues

I've been experiencing 3 different issues related to the SQL endpoint not being refreshed quickly enough or at all when the undelying delta table is updated with new data. 1) A pipeline will fail wh...
pipelines
  • dzav's avatar
    dzav
    2 years ago

    frithjof_v , thank you! Looks like that was the optimal solution. I now how the following pattern:

    1) Load the bronze lakehouse

    2) A copy task to load the bronze warehouse from the bronze lakehouse

    3) A stored procedure to copy from the bronze warehouse to the silver warehouse

     

    I don't know how scalabale this is because we have over a hundred tables from Synapse Link and adding tables to a copy task is a bit onerous, but for now this seems to be working.

dzav's avatar
dzav
Icon for Advocate III rankAdvocate III
2 years ago

Interesting! I will test it out!

dzav's avatar
dzav
Icon for Advocate III rankAdvocate III
2 years ago

frithjof_v , thank you! Looks like that was the optimal solution. I now how the following pattern:

1) Load the bronze lakehouse

2) A copy task to load the bronze warehouse from the bronze lakehouse

3) A stored procedure to copy from the bronze warehouse to the silver warehouse

 

I don't know how scalabale this is because we have over a hundred tables from Synapse Link and adding tables to a copy task is a bit onerous, but for now this seems to be working.