Forum Discussion
V-Order Functionality in Fabric Notebook
DennesTorres to me it seems like a bug , when tables are not optimized at session level while they are at dataframe write level ( by adding "parquet.vorder.enabled ","true") even after that there seems no signs that it has been optimized in metadata/ logs not even at the time of its first creation
After doing checks at _delta logs and in the metadata and even table properties
There is no v order which has been present after checking by all three methods , i
am not sure if any other method is also availiable, if not then to me it sounds more like a bug
Hi,
But or missing feature, I don't know. When the optimization is on session level, it's strange to me that the 'V-ORDER' appears only as TAG, a property not intended to mean something so important as the optmization format of the file (I think so, what are your thoughts?).When optimizing on the write level, the optmization is designed to affect specific parquet files, so it's not included on the TAG of the table and since it was not included on metadata anyway, we are left with no way to confirm the optmization. There is some logic, but it's still a missing point.
We end up with many questions: Why does it only appear on the TAG?
How to identify the optimization on individual parquet files?
What about OpmizeWrite, for example, which appears no where ?
Kind Regards,
DennesDennesTorres based on this official doc i can say its indeed optimization at parquet level files
https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksqlWhen i turn them on on session level i see them applied on parquet files metatata using pyarrow ( above code)
Also as we see tag in delta logs level as well ..my guess tag in delta log would be there for some reason
the accurate way i understad is to check at the parquet's metatadata level , however in my mentioned issue when i turn them off at session level and try to control at dataframe write level i do not see expected behavior in parquet's metata
Hi,
I tried your script. It will help me a lot in other purposes, but I confirm your result: If the optmization doesn't happen on session level there is no sign of any metadata pointing to the optmization.
To be absolutely sure about the result, I completed a test on the file level as well. The original article mentions only the parquet files affected by the write operation would be optmized.
But even on file level, there is no metadata pointing to any optmization.
The script I tried, a variation from yours, is located below. What's the link to the issue you registered?import pyarrow.dataset as pqdef print_metadata(delta_file_path):# Print schema metadataprint("\nSchema Metadata:")print("--------------------")schema_metadata = pq.dataset(delta_file_path).schema.metadataif schema_metadata:for key, value in schema_metadata.items():print(f"{key.decode('utf-8')}: {value.decode('utf-8')}")else:print("No schema metadata found.")# Test the function with a path to your delta file#full_tables = ['part-00000-cea68c9b-4b96-4955-a1b4-1147b89e8a15-c000.snappy.parquet','part-00001-9d86a807-c4b6-4c33-897e-eeedcfed4ca9-c000.snappy.parquet','part-00002-694dfef3-bea1-4c8e-99bb-8c45a926e413-c000.snappy.parquet','part-00003-dcbf684b-a2b0-4b20-8a63-c962f234e674-c000.snappy.parquet','part-00004-ecb43c9b-1ffb-46a9-9289-50ac5d02683f-c000.snappy.parquet','part-00005-dd4030dc-00be-43bd-873f-374393545d8a-c000.snappy.parquet']for table in full_tables:print_metadata('//lakehouse/default/Tables/dimension_city/' + table)Kind Regards,Dennes
