Forum Discussion

p_da's avatar
p_da
Helper I
1 year ago
Solved

Lakehouse SQL endpoint

Hi all,   I have a quick question. I know that MS is already on track of the current issue with Lakehouse where SQL endpoint is taking some time to sync the metadata with lakehouse. Does we have an...
data engineering
lakehouse
  • AlexanderPowBI's avatar
    AlexanderPowBI
    1 year ago

    As my issue is ETL / pipeline related, neither of that helps. 

    What I did now as a soloution until MS releases the API which frithjof_v mentions, is to use the "unsupported script" to update SQL endpoints written by somone at MSFT found here: 

    github 

    What I did, if someone is interested to:

    1. Rewrote the script to be a bit more robust, with some error handling etc. 
    2. Created it as a wheel, uploaded in a shared lakehouse, call it "own modules". 
    3. Shortcuted into "own modules" in every lakehouse where a notebook updates the data & a sql endpoint needs a refresh.
    4. At end of each notebook which updates lakehouses, i run the script with error handling & logging, so if the unsupported APIs in the script gets changed/stops working, it won't affect the pipeline run, but I will know from logs. 
    5. Wait for the official API to come. 

    Best regards,

    Alexander