Forum Discussion
V-Order Functionality in Fabric Notebook
Hi,
But or missing feature, I don't know. When the optimization is on session level, it's strange to me that the 'V-ORDER' appears only as TAG, a property not intended to mean something so important as the optmization format of the file (I think so, what are your thoughts?).
When optimizing on the write level, the optmization is designed to affect specific parquet files, so it's not included on the TAG of the table and since it was not included on metadata anyway, we are left with no way to confirm the optmization. There is some logic, but it's still a missing point.
We end up with many questions: Why does it only appear on the TAG?
How to identify the optimization on individual parquet files?
What about OpmizeWrite, for example, which appears no where ?
Kind Regards,
Dennes
DennesTorres based on this official doc i can say its indeed optimization at parquet level files
https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksql
When i turn them on on session level i see them applied on parquet files metatata using pyarrow ( above code)
Also as we see tag in delta logs level as well ..my guess tag in delta log would be there for some reason
the accurate way i understad is to check at the parquet's metatadata level , however in my mentioned issue when i turn them off at session level and try to control at dataframe write level i do not see expected behavior in parquet's metata