Forum Discussion
Delta Tables in LakeHouse missing in SQL Endpoint
I have a number of tables loaded from a large set of incremetal json files. Once my load completed I noticed that some of the tables I can see in the Lakehouse are not showing up in SQL Endpoint. Ive...
Thanks Anonymous
I note that after the weekend that the missing tables have now showed up in SQL Endpoint. However not sure that the business is prepared to wait several days for data to be available after ingestion.
How are you loading these tables into lakehouse?
Im creating tables using:(df.write.mode(mode).partitionBy("file_name","year", "month" ,"day").format("delta").save(deltaTableName))note: mode = "overwrite"
How large is the volume of the tables?
Im new to notebooks/pyspark/delta tables havng come from a sql based background, so not sure how to calculate that in terms of what your asking?2) The issue can be due to a schema mismatch. You might need to update the schema of your Delta tables or fix any schema mismatches to ensure that all tables are included in the INFORMATION_SCHEMA.TABLES.
In terms of the schema, in my data load code, for the initial load the code reads 2 input folders into two df's, one containing "current files" and the other "archived files". The code then unions the complete list of columns from both df's, creating a common cols list and then ensures both data frames have the same columns in the same order. The df's are then unioned and the code then applies rules to force data types on specific columns. i.e the schem infered from each df can vary so I have rules like "if column_name endswith "_ID" then IntergerType() or if column_)name = "LAST_MODIFIED_DATE" then TimestampType() etc etc.Kind RegardsTim
The load "dbo" schema preview in lakehouse is causing a failure to load tables in the SQL Endpoint. Make sure not to click on this preview feature...it is buggy.