Forum Discussion

puneetvijwani's avatar
puneetvijwani
Resolver IV
3 years ago

V-Order Functionality in Fabric Notebook

I've been experiencing an issue with my scripts over Fabric Notebooks in terms of optimization . I've been trying to control the V-Order functionality in data loading processes. According to the docu...
data engineering
puneetvijwani's avatar
puneetvijwani
Resolver IV
3 years ago

DennesTorres 
i am checking in one lake explorer mannualy _delta_log files if VOrder Tag is present and is set to true 

You can also write spark code to check the delta log details from notebook 
here is my code to read metadata of the table 

 

tablebasepath="your path to the table "

tablename=f'{tablebasepath}/_delta_log'
# Get a list of all JSON files in the _delta_log directory
log_files = [file.path for file in mssparkutils.fs.ls(tablename) if file.name.endswith(".json")]
# Check if there are any log files
if log_files:
    # Read the first log file
    data = mssparkutils.fs.head(log_files[0])
   
    # Print the contents of the file
    print(data)
else:
    print("No log files found.")

However thanks for redirecting me to the issues page i should rather post this is an issue
DennesTorres's avatar
DennesTorres
Impactful Individual
3 years ago

Hi,

 

I tested your script. All the tables I have were marked with V-ORDER, but what caught my attention was that the V-ORDER was only a TAG, it was not on the metadata. Is this correct?

I managed to turn off V-ORDER optimization on session level and the TAG disappeared. But after that I repeated your results: Trying to enable vorder for one specific write operation doesn't bring the TAG back.

One detail about this and the article you are using is one mention to the fact that the write configuration will affect only the parquet files involved in the operation, not the entire table. Could this result in a table with mixed parquet files, some with the optimization and some not? Could this explain why the V-Order doesn't appear on table level ?