Forum Discussion

puneetvijwani's avatar
puneetvijwani
Resolver IV
3 years ago

V-Order Functionality in Fabric Notebook

I've been experiencing an issue with my scripts over Fabric Notebooks in terms of optimization . I've been trying to control the V-Order functionality in data loading processes. According to the docu...
data engineering
DennesTorres's avatar
DennesTorres
Impactful Individual
3 years ago

Hi,

About this:

"But despite this, both instances in the script seem to be executing with V-Order disabled for both of them in a single session"

How do you know the v-order was disabled?

"We've also looked through our delta logs for both files in Tables and in Files folder, but there are no traces of the Vorder tag to be found, which is puzzling and contrary."

Could you give more details about this?

 

I have been working on a related challenge: How to identify if an existing table was created with v-order enabled or not?

Here is a thread about my investigation: https://community.fabric.microsoft.com/t5/General-Discussion/How-to-list-Table-Properties/m-p/3371301

Another one which may or may not be related: https://community.fabric.microsoft.com/t5/Issues/Workspace-level-boolean-spark-configurations-appear-empty-when/idc-p/3373826

Kind Regards,

 

Dennes

puneetvijwani's avatar
puneetvijwani
Resolver IV
3 years ago

DennesTorres 
i am checking in one lake explorer mannualy _delta_log files if VOrder Tag is present and is set to true 

You can also write spark code to check the delta log details from notebook 
here is my code to read metadata of the table 

 

tablebasepath="your path to the table "

tablename=f'{tablebasepath}/_delta_log'
# Get a list of all JSON files in the _delta_log directory
log_files = [file.path for file in mssparkutils.fs.ls(tablename) if file.name.endswith(".json")]
# Check if there are any log files
if log_files:
    # Read the first log file
    data = mssparkutils.fs.head(log_files[0])
   
    # Print the contents of the file
    print(data)
else:
    print("No log files found.")

However thanks for redirecting me to the issues page i should rather post this is an issue
  • DennesTorres's avatar
    DennesTorres
    Impactful Individual
    3 years ago

    Hi,

     

    I tested your script. All the tables I have were marked with V-ORDER, but what caught my attention was that the V-ORDER was only a TAG, it was not on the metadata. Is this correct?

    I managed to turn off V-ORDER optimization on session level and the TAG disappeared. But after that I repeated your results: Trying to enable vorder for one specific write operation doesn't bring the TAG back.

    One detail about this and the article you are using is one mention to the fact that the write configuration will affect only the parquet files involved in the operation, not the entire table. Could this result in a table with mixed parquet files, some with the optimization and some not? Could this explain why the V-Order doesn't appear on table level ?