topic Re: Lakehouse with Schema Bug - Not generating proper metadata in Data Engineering

Lakehouse with Schema Bug - Not generating proper metadata

MattSB — Tue, 31 Dec 2024 09:10:34 GMT

HI ive ran into a problem while testing the Lakehouse with schema. The issue appears that the creation of the deltatable is not populating the standard meta data correctly

Issue

- lakehouse called 'bronze' with a schema called `rdm`.

- Notebook requesting data from gov.uk API adding to a datframe and loading to a delta table.

- Issues identified

In the properties pane Format is not populated.
Running `spark.catalog.listTables()` returns an empty list
Running `spark.sql("show tables")` returns an empty data frame
Running `DESCRIBE DETAIL rdm.bank_holidays_eng` returns an odd name but is returning the format of delta
Running DESCRIBE DETAIL delta.`Tables/rdm/bank_holidays_eng` returns NULL for name

Test Scenario

- Notebook loading

# load the requests library import requests # define the url url = "https://www.gov.uk/bank-holidays/england-and-wales.json" # make a GET request response = requests.get(url) # Process the response json response_data = response.json() ## Create the data frame ## # Load modules from pyspark.sql.types import StructType, StructField, StringType, DateType from datetime import datetime # Convert date strings to date objects, handling blank strings def convert_date(date_str): return datetime.strptime(date_str, '%Y-%m-%d').date() if date_str else None # Extract the list of results parsed_data = [(item['title'], convert_date(item['date'])) for item in response_data['events']] # Define the schema schema = StructType([ StructField("holiday", StringType(), True, {"comment": "Name of holiday"}), StructField("date", DateType(), True, {"comment": "Date of holiday"}) ]) # Create DataFrame df = spark.createDataFrame(parsed_data , schema ) # Order the results df = df.orderBy(df["date"].asc()) # Show the dataframe df.show() ## Store data in delta table ## delta_table_path = "bronze.rdm.bank_holidays_eng" #also tried using the table path and same behaviour "Tables/rdm/bank_holidays_eng" # Write the DataFrame to a Delta table df.write.format("delta").mode("overwrite").saveAsTable(delta_table_path)

Any advice would be appreciated.

Re: Lakehouse with Schema Bug - Not generating proper metadata

v-nmadadi-msft — Tue, 31 Dec 2024 14:37:55 GMT

Hi MattSB ,
Thanks for reaching out to the Microsoft fabric community forum.

Currently as the lakehouse schemas is still in public preview some of the features are not yet supported. One of them is adding a delta table in our desired schema. Currently we can load df into the default schema.

But we can then move the delta table to the schema we want.

Once the delta table is created in default schema we can just drag and drop it from default schema to the one we want

Alterantively you can also use DDL commands (Alter tables) in SQL endpoint to move the delta table

Did I answer your question? Mark my post as a solution, this will help others!

If my response(s) assisted you in any way, don't forget to give "Kudos"

Thanks and Regards

Re: Lakehouse with Schema Bug - Not generating proper metadata

MattSB — Tue, 31 Dec 2024 14:57:47 GMT

HI nithin,

Thanks for confirming, i was sort of hoping after 4 months in preview the schema functionality would be closer to production by now.

Can you confirm if I use your suggested process will appends and overwrites work still?

Re: Lakehouse with Schema Bug - Not generating proper metadata

v-nmadadi-msft — Wed, 01 Jan 2025 17:05:40 GMT

Hi MattSB ,
you can still append or overwrite to the table in your default schema and then drag and drop the table to a different schema.

But if you are trying to append or overwrite to a table in different schema using SQL endpoint( or Spark SQL) would be the better option.
Thanks and regards

Re: Lakehouse with Schema Bug - Not generating proper metadata

v-nmadadi-msft — Mon, 06 Jan 2025 04:54:26 GMT

Hi MattSB

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.

Re: Lakehouse with Schema Bug - Not generating proper metadata

v-nmadadi-msft — Wed, 15 Jan 2025 10:14:00 GMT

Hi MattSB

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.

Re: Lakehouse with Schema Bug - Not generating proper metadata

v-nmadadi-msft — Wed, 22 Jan 2025 05:01:04 GMT

Hi MattSB

I wanted to confirm if the solution provided was helpful and resolved your concerns. If you have any further questions or issues, please feel free to reach out to us, also if this post helps, then please give us Kudos and consider Accept it as a solution to help the other members find it more quickly.

Thank you.