Forum Discussion

puneetvijwani's avatar
puneetvijwani
Resolver IV
3 years ago

V-Order Functionality in Fabric Notebook

I've been experiencing an issue with my scripts over Fabric Notebooks in terms of optimization . I've been trying to control the V-Order functionality in data loading processes. According to the docu...
data engineering
puneetvijwani's avatar
puneetvijwani
Resolver IV
3 years ago

DennesTorres  to me it seems like a bug , when tables are not optimized at session level while they are at dataframe write level ( by adding  "parquet.vorder.enabled ","true")  even after that there seems no signs that it has been optimized in metadata/ logs not even at the time of its first creation
After doing  checks at  _delta logs and in the metadata and even table properties

There is no v order which has been present after checking by all three methods , i

am not sure if any other method is also availiable, if not then to me it sounds more like a bug 

DennesTorres's avatar
DennesTorres
Impactful Individual
3 years ago

Hi,

 

I tried your script. It will help me a lot in other purposes, but I confirm your result: If the optmization doesn't happen on session level there is no sign of any metadata pointing to the optmization.

To be absolutely sure about the result, I completed a test on the file level as well. The original article mentions only the parquet files affected by the write operation would be optmized. 

But even on file level, there is no metadata pointing to any optmization.

The script I tried, a variation from yours, is located below. What's the link to the issue you registered?

import pyarrow.dataset as pq

def print_metadata(delta_file_path):
   
    # Print schema metadata
    print("\nSchema Metadata:")
    print("--------------------")
    schema_metadata = pq.dataset(delta_file_path).schema.metadata
    if schema_metadata:
        for key, value in schema_metadata.items():
            print(f"{key.decode('utf-8')}: {value.decode('utf-8')}")
    else:
        print("No schema metadata found.")
   
    # Test the function with a path to your delta file#

full_tables = [
    'part-00000-cea68c9b-4b96-4955-a1b4-1147b89e8a15-c000.snappy.parquet',
    'part-00001-9d86a807-c4b6-4c33-897e-eeedcfed4ca9-c000.snappy.parquet',
    'part-00002-694dfef3-bea1-4c8e-99bb-8c45a926e413-c000.snappy.parquet',
    'part-00003-dcbf684b-a2b0-4b20-8a63-c962f234e674-c000.snappy.parquet',
    'part-00004-ecb43c9b-1ffb-46a9-9289-50ac5d02683f-c000.snappy.parquet',
    'part-00005-dd4030dc-00be-43bd-873f-374393545d8a-c000.snappy.parquet'
    ]

for table in full_tables:
     print_metadata('//lakehouse/default/Tables/dimension_city/' + table)
 
Kind Regards,
 
Dennes