Forum Discussion

puneetvijwani's avatar
puneetvijwani
Resolver IV
3 years ago

V-Order Functionality in Fabric Notebook

I've been experiencing an issue with my scripts over Fabric Notebooks in terms of optimization . I've been trying to control the V-Order functionality in data loading processes. According to the docu...
data engineering
puneetvijwani's avatar
puneetvijwani
Resolver IV
3 years ago

DennesTorres I checked at metadata level of parquet files using pyarrow and printing dataset level schema for the table using below function .. i still dont see any metatadata related to Vorder ..not sure what i am missing ..can you test this at your end and see do you also seeing same behaviour ??

import pyarrow.dataset as pq

def print_metadata(delta_file_path):
   
    # Print schema metadata
    print("\nSchema Metadata:")
    print("--------------------")
    schema_metadata = pq.dataset(delta_file_path).schema.metadata
    if schema_metadata:
        for key, value in schema_metadata.items():
            print(f"{key.decode('utf-8')}: {value.decode('utf-8')}")
    else:
        print("No schema metadata found.")
   
    # Test the function with a path to your delta file
# print_metadata('//lakehouse/default/Tables/table_name')



  • DennesTorres's avatar
    DennesTorres
    Impactful Individual
    3 years ago

    Hi,

    I will check.

    But, this may be related to my previous comment: On the tables optimized by session level configuration, the V-ORDER optimization appears only as a TAG, it doesn't appear as metadata and I don't know exactly why. Would this mean the table is not optimized, or we need a different way to identify it is optmized ?

     

    When trying the optimization on the write, instead of session level, the TAG doesn't appear as well.

    Kind Regards,

     

    Dennes