Forum Discussion
Lakehouse SQL endpoint
As my issue is ETL / pipeline related, neither of that helps.
What I did now as a soloution until MS releases the API which frithjof_v mentions, is to use the "unsupported script" to update SQL endpoints written by somone at MSFT found here:
What I did, if someone is interested to:
- Rewrote the script to be a bit more robust, with some error handling etc.
- Created it as a wheel, uploaded in a shared lakehouse, call it "own modules".
- Shortcuted into "own modules" in every lakehouse where a notebook updates the data & a sql endpoint needs a refresh.
- At end of each notebook which updates lakehouses, i run the script with error handling & logging, so if the unsupported APIs in the script gets changed/stops working, it won't affect the pipeline run, but I will know from logs.
- Wait for the official API to come.
Best regards,
Alexander
Hi,
I was about to create a topic on what I believe is a similar issue.
In my pipeline, I have an audit step that utilizes SQL endpoints for auditing. However, these checks often fail falsely because the queries seem to be based on outdated data. They do not reflect updates that have just occurred in the pipeline. For example, if Table X initially has 100 records and the pipeline inserts 50 more, the table count still shows 100 instead of 150.
I believe this is related to the issue being discussed. Is the only solution coming up the one refered about API update trigger?
//Alexander
Hi AlexanderPowBI ,
Thanks for bringing this up. It looks like the issue you're facing is related to the ongoing delay in SQL Endpoint metadata sync in Lakehouse. As mentioned earlier, SQL Endpoints may take a bit of time to reflect recent changes, which can lead to outdated query results.
Here are a couple of alternative approaches that might help.
-
If you're using Power BI, switching to Direct Lake mode can bypass SQL Endpoints and provide you with near real-time data access.
-
Instead of relying only on SQL Endpoints, consider querying your Lakehouse data using Notebooks or the Lakehouse REST API to get the most up-to-date results.
Hope this helps! If you need any further clarification, feel free to ask.
If my response solved your query, please mark it as the Accepted solution to help others find it easily!
And if my answer was helpful, I'd really appreciate a 'Kudos'.
As my issue is ETL / pipeline related, neither of that helps.
What I did now as a soloution until MS releases the API which frithjof_v mentions, is to use the "unsupported script" to update SQL endpoints written by somone at MSFT found here:
What I did, if someone is interested to:
- Rewrote the script to be a bit more robust, with some error handling etc.
- Created it as a wheel, uploaded in a shared lakehouse, call it "own modules".
- Shortcuted into "own modules" in every lakehouse where a notebook updates the data & a sql endpoint needs a refresh.
- At end of each notebook which updates lakehouses, i run the script with error handling & logging, so if the unsupported APIs in the script gets changed/stops working, it won't affect the pipeline run, but I will know from logs.
- Wait for the official API to come.
Best regards,
Alexander
Hi AlexanderPowBI ,
Thank you for providing your insights and workaround. For further assistance, please continue the discussion in the Fabric Community.
Regards,
Yugandhar.