Forum Discussion

puneetvijwani's avatar
puneetvijwani
Resolver IV
3 years ago

V-Order Functionality in Fabric Notebook

I've been experiencing an issue with my scripts over Fabric Notebooks in terms of optimization . I've been trying to control the V-Order functionality in data loading processes. According to the docu...
data engineering
puneetvijwani's avatar
puneetvijwani
Resolver IV
3 years ago

DennesTorres  to me it seems like a bug , when tables are not optimized at session level while they are at dataframe write level ( by adding  "parquet.vorder.enabled ","true")  even after that there seems no signs that it has been optimized in metadata/ logs not even at the time of its first creation
After doing  checks at  _delta logs and in the metadata and even table properties

There is no v order which has been present after checking by all three methods , i

am not sure if any other method is also availiable, if not then to me it sounds more like a bug 

DennesTorres's avatar
DennesTorres
Power Participant
3 years ago

Hi,

But or missing feature, I don't know. When the optimization is on session level, it's strange to me that the 'V-ORDER' appears only as TAG, a property not intended to mean something so important as the optmization format of the file (I think so, what are your thoughts?).

 

When optimizing on the write level, the optmization is designed to affect specific parquet files, so it's not included on the TAG of the table and since it was not included on metadata anyway, we are left with no way to confirm the optmization. There is some logic, but it's still a missing point.

We end up with many questions: Why does it only appear on the TAG?

How to identify the optimization on individual parquet files?

What about OpmizeWrite, for example, which appears no where ?

Kind Regards,

Dennes

  • puneetvijwani's avatar
    puneetvijwani
    Resolver IV
    3 years ago

    DennesTorres based on this official doc i can say its indeed  optimization at parquet level files 
    https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksql

     

    When i turn them on on session level i see them applied on parquet files metatata using pyarrow ( above code)

     

    Also as we see tag in delta logs level as well ..my guess tag in delta log would be there for some reason 

    the accurate way i understad is to check at the parquet's metatadata level , however in my mentioned issue when i turn them off at session level and try to control at dataframe write level i do not see expected behavior in parquet's metata