Lakehouse with Schema Bug - Not generating proper metadata

[MattSB]

HI ive ran into a problem while testing the Lakehouse with schema. The issue appears that the creation of the deltatable is not populating the standard meta data correctly 

Issue

- lakehouse called 'bronze' with a schema called `rdm`.

- Notebook requesting data from gov.uk API adding to a datframe and loading to a delta table.

- Issues identified

1. In the properties pane Format is not  populated.
   
2.  Running `spark.catalog.listTables()` returns an empty list
3.  Running `spark.sql("show tables")` returns an empty data frame
4. Running `DESCRIBE DETAIL rdm.bank_holidays_eng` returns an odd name but is returning the format of delta
   

5. Running DESCRIBE DETAIL delta.`Tables/rdm/bank_holidays_eng` returns NULL for name 
   

   

Test Scenario

- Notebook loading 

# load the requests library
import requests

# define the url
url = "https://www.gov.uk/bank-holidays/england-and-wales.json"

# make a GET request 
response = requests.get(url)

# Process the response json 
response_data = response.json()

## Create the data frame ##

# Load modules
from pyspark.sql.types import StructType, StructField, StringType, DateType
from datetime import datetime

# Convert date strings to date objects, handling blank strings
def convert_date(date_str):
    return datetime.strptime(date_str, '%Y-%m-%d').date() if date_str else None

# Extract the list of results
parsed_data = [(item['title'], convert_date(item['date'])) for item in response_data['events']]

# Define the schema
schema = StructType([
    StructField("holiday", StringType(), True, {"comment": "Name of holiday"}),
    StructField("date", DateType(), True, {"comment": "Date of holiday"})
])

# Create DataFrame 
df = spark.createDataFrame(parsed_data , schema )

# Order the results
df = df.orderBy(df["date"].asc())

# Show the dataframe
df.show()

## Store data in delta table ##

delta_table_path = "bronze.rdm.bank_holidays_eng" 
#also tried using the table path and same behaviour "Tables/rdm/bank_holidays_eng"

# Write the DataFrame to a Delta table
df.write.format("delta").mode("overwrite").saveAsTable(delta_table_path)

Any advice would be appreciated.

[v-nmadadi-msft]

Hi @MattSB 

I wanted to confirm if the solution provided was helpful and resolved your concerns. If you have any further questions or issues, please feel free to reach out to us, also if this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thank you.

[v-nmadadi-msft]

Hi @MattSB 

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.

[MattSB]

HI nithin,

Thanks for confirming, i was sort of hoping after 4 months in preview the schema functionality would be closer to production by now. 

Can you confirm if I use your suggested process will appends and overwrites work still? 

[v-nmadadi-msft]

Hi @MattSB ,
you can still append or overwrite to the table in your default schema and then drag and drop the table to a different schema.

But if you  are trying to append or overwrite to a table in different schema using SQL endpoint( or Spark SQL) would be the better option.
Thanks and regards

[v-nmadadi-msft · Accepted solution]

Hi @MattSB  ,
Thanks for reaching out to the Microsoft fabric community forum.

Currently as the lakehouse schemas is still in public preview some of the features are not yet supported. One of them is adding a delta table in our desired schema. Currently we can load df into the default schema.

 But we can then move the delta table to the schema we want.

Once the delta table is created in default schema we can just drag and drop it from default schema to the one we want

Alterantively you can also use DDL commands (Alter tables) in SQL endpoint to move the delta table

Did I answer your question? Mark my post as a solution, this will help others!

If my response(s) assisted you in any way, don't forget to give "Kudos"

Thanks and Regards

[v-nmadadi-msft]

Hi @MattSB 

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.