Hi,
I'm making some experiments using the OPTIMIZE statement and VORDER configuration.
Basically, I create a table without the VORDER configuration and use OPTMIZE to apply the VORDER.
This should work according to this documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparks...
I few days (one week?) ago, I recorded a video executing these tasks and everything worked correctly.
Today I'm trying to execute these tasks again, while writing a blog post, and it's failing.
Was there any change in Fabric which caused this? Why the change is in conflict with the documentation?
Before:
The statement was accepted, it didn't caused any error:
Once executed, the VORDER appeared in the table metadata.
Now:
VORDER is pointed as an error by the intellisense:
It can be executed without errors, but the table metadata doesn't change.
Is this an issue, or is this a change of behaviour which is not in sync with the documentation yet ?
By the way:
I'm reading the metadata of the table, using the following code:
import pyarrow.dataset as pq
import os
def show_metadata(delta_file_path,tablename):
# Print schema metadata
print(f"\nSchema Properties for : {tablename}")
print("--------------------")
schema_properties = pq.dataset(delta_file_path).schema.metadata
if schema_properties:
for key, value in schema_properties.items():
print(f"{key.decode('utf-8')}: {value.decode('utf-8')}")
else:
print("No schema properties found.")
# Test the function with a path to your delta file#
full_tables = os.listdir('/lakehouse/default/Tables')
for table in full_tables:
show_metadata('//lakehouse/default/Tables/' + table,table)
The ALTER TABLE statement changes the TBLPROPERTIES, for future writes on the table. Tables with the metadata defined, don't have the TBLPROPERTIES defined. In fact, the TBLPROPERTIES is not default. Tables with the TBLPROPERTIES defined may not have the metadata defined.
I tested the ALTER TABLE and executed and OPTIMIZE after it. The table got the TBLPROPERTY for VORDER, but not the metadata for VORDER, making this very confuse.
Kind Regards,
I tested the ALTER TABLE and executed and OPTIMIZE after it. The table got the TBLPROPERTY for VORDER, but not the metadata for VORDER, making this very confuse.
Kind Regards,
Dennes
Kind Regards,
Dennes
