Forum Discussion
Warehouse data not refreshing when pulling from Lakehouse
Hi marius1106,
Based on my understanding, I recommend first copying the raw data into a Warehouse table and then applying T-SQL transformations using views or additional logic. This approach helps avoid delays associated with the SQL endpoint and ensures that you are working with the most recent data in a stable environment that is optimised for transformations.
Please note that Notebooks may fail, as T-SQL Notebooks rely on the Lakehouse SQL analytics endpoint, which can experience metadata synchronisation delays. These delays may affect timely access to data, even for existing tables.
Other alternatives, such as introducing wait or delay logic, may provide some relief but are not entirely reliable. Similarly, using a view over Lakehouse tables is a potential solution, but it may also be impacted by the same metadata sync issues.
If you find this response helpful, we kindly request you to mark it as the accepted solution and consider giving kudos. This will help other community members who may have similar queries.
Thank you.
Thankyou, Akash_Varuna , ObungiNiels ,for your response.
Hi marius1106,
Based on my understanding, you should not face the same issue when pulling data from a Lakehouse using a Data Pipeline. Unlike the T-SQL notebook, which depends on the SQL Analytics Endpoint (where delays may occur due to metadata syncing), a Data Pipeline’s Copy activity directly accesses the Lakehouse Delta tables via OneLake. This eliminates the sync delay and ensures that the Warehouse receives the latest data.
Additionally, please refer to the links below for more details:
Better together - the lakehouse and warehouse - Microsoft Fabric | Microsoft Learn
SQL analytics endpoint performance considerations - Microsoft Fabric | Microsoft Learn
If you find our response helpful, kindly mark it as the accepted solution and give kudos. This will help other community members with similar queries.
Thank you.
Hi v-pnaroju-msft ,
thank you for your response. Unfortunately I can't migrate all my TSQL-Notebooks into Pipelines so that it all works with the Lakehouse-delay.
Is there another possibility to implement my TSQL-Transformations and save it into a Table/View like in the Code above and get the actual data from a Lakehouse?
I think of workarounds like:
A) First copy all the raw data to a Warehouse and than transform it there in a View-definiton
B) Add something like a time.sleep() Command or a refresh-command into the notebook/pipeline
C) Put the code above into a View-definiton instead of a TSQL-Notebook
Is there one working workaround in my 3 ideas? And how would you implement it then?
Thank you!