Forum Discussion
Delta Tables in LakeHouse missing in SQL Endpoint
Thanks Anonymous
I note that after the weekend that the missing tables have now showed up in SQL Endpoint. However not sure that the business is prepared to wait several days for data to be available after ingestion.
How are you loading these tables into lakehouse?
Im creating tables using:(df.write.mode(mode).partitionBy("file_name","year", "month" ,"day").format("delta").save(deltaTableName))note: mode = "overwrite"
How large is the volume of the tables?
Im new to notebooks/pyspark/delta tables havng come from a sql based background, so not sure how to calculate that in terms of what your asking?2) The issue can be due to a schema mismatch. You might need to update the schema of your Delta tables or fix any schema mismatches to ensure that all tables are included in the INFORMATION_SCHEMA.TABLES.
In terms of the schema, in my data load code, for the initial load the code reads 2 input folders into two df's, one containing "current files" and the other "archived files". The code then unions the complete list of columns from both df's, creating a common cols list and then ensures both data frames have the same columns in the same order. The df's are then unioned and the code then applies rules to force data types on specific columns. i.e the schem infered from each df can vary so I have rules like "if column_name endswith "_ID" then IntergerType() or if column_)name = "LAST_MODIFIED_DATE" then TimestampType() etc etc.Kind RegardsTim
Thanks Anonymous
I note that after the weekend that the missing tables have now showed up in SQL Endpoint. However not sure that the business is prepared to wait several days for data to be available after ingestion.
How are you loading these tables into lakehouse?
Im creating tables using:
How large is the volume of the tables?
Im new to notebooks/pyspark/delta tables havng come from a sql based background, so not sure how to calculate that in terms of what your asking?
In terms of the schema, in my data load code, for the initial load the code reads 2 input folders into two df's, one containing "current files" and the other "archived files". The code then unions the complete list of columns from both df's, creating a common cols list and then ensures both data frames have the same columns in the same order. The df's are then unioned and the code then applies rules to force data types on specific columns. i.e the schem infered from each df can vary so I have rules like "if column_name endswith "_ID" then IntergerType() or if column_)name = "LAST_MODIFIED_DATE" then TimestampType() etc etc.
Hi Anonymous
Thanks for providing the details. As you are able to see the tables now, this might be a temporary glitch. Please let me know if you face the issue again.
Thanks
I'm having the same issue, I can see the newly export delta table in Lakehouse, but not in SQL end point. I have retried refreshing table and reconnecting with no luck.
Hi Jeanxyz,
I found this blog post that triggers the refresh of a lakehouse with a Python script. It works like a charm for me. I had the problem that the lakehouse had a lag of 30+ minutes. This Python script (which I used in a notebook) reduced it to 6 minutes and it waits until the sql endpoint is refreshed. So you are sure that other activities which are executed on completion of the Python notebook will have the refreshed sql endpoint.
https://www.obvience.com/blog/fix-sql-analytics-endpoint-sync-issues-in-microsoft-fabric-data-not-sh...
