Forum Discussion

Bill_J99294's avatar
Bill_J99294
Regular Visitor
1 year ago
Solved

Fabric Lakehouse SQL Endpoint Automatic Refresh

Hello,    Recently we have noticed that the automatic metadata discovery for our lakehouse has not been working. This has resulted in us manually refreshing the SQL endpoint for our Lakehouse each ...
data warehouse
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hi Bill_J99294 ,

     

    You can create a notebook that connects to the lakehouse corresponding to your SQL endpoint by entering the following command in the cell:

    from pyspark.sql import SparkSession

# create Spark session
spark = SparkSession.builder \
    .appName(“Refresh SQL Endpoint Metadata”) \
    .getOrCreate()

# refresh
spark.sql(“REFRESH TABLE salesorders”)

print(“Metadata refresh triggered successfully.”)

     

    Set up a daily refresh in your notebook settings so that this refresh command is executed every day.

     

    If you have any other questions please feel free to contact me.

     

    Best Regards,
    Yang
    Community Support Team

     

    If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

amokhalgay's avatar
amokhalgay
Regular Visitor
1 year ago

Anonymous Can we refresh the whole Lakehouse SQL Endpoint using the spark notebook, as the command it looks like it is just going to refresh the table? 

Our customers are facing issue in which when we upload the Delta File in Lakehouse after that table is not visible even after waiting for 5 mins. So is there a way we can do via notebook, please suggest.

  • jankuc's avatar
    jankuc
    Advocate I
    1 year ago

    When we discussed this with msft, they said we should run a dummy query (e.g. SELECT count(*) ...) and then wait a bit (they didn't say how long but try to experiment with 30, 60 seconds). This should start the serverless sql endpoint and after that run the automatic refresh. 
    We've also tried to call the undocummented API, which was working for us. https://gist.github.com/MarkPryceMaherMSFT/853cdc0d9d421482814b8195aba55434