Forum Discussion

DennesTorres's avatar
DennesTorres
Power Participant
2 years ago

OPTIMIZE and VORDER

Hi,

I'm making some experiments using the OPTIMIZE statement and VORDER configuration.

 

Basically, I create a table without the VORDER configuration and use OPTMIZE to apply the VORDER.

 

This should work according to this documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksql 

 

I few days (one week?) ago, I recorded a video executing these tasks and everything worked correctly. 

Today I'm trying to execute these tasks again, while writing a blog post, and it's failing. 

Was there any change in Fabric which caused this? Why the change is in conflict with the documentation?

Before:

 

The statement was accepted, it didn't caused any error:


 

Once executed, the VORDER appeared in the table metadata.

Now:

 

VORDER is pointed as an error by the intellisense:

 

It can be executed without errors, but the table metadata doesn't change.

Is this an issue, or is this a change of behaviour which is not in sync with the documentation yet ?

 

By the way:

I'm reading the metadata of the table, using the following code:

 

import pyarrow.dataset as pq
import os

def show_metadata(delta_file_path,tablename):
   
    # Print schema metadata
    print(f"\nSchema Properties for : {tablename}")
    print("--------------------")
    schema_properties = pq.dataset(delta_file_path).schema.metadata
    if schema_properties:
        for key, value in schema_properties.items():
            print(f"{key.decode('utf-8')}: {value.decode('utf-8')}")
    else:
        print("No schema properties found.")
   
    # Test the function with a path to your delta file#

full_tables = os.listdir('/lakehouse/default/Tables')

for table in full_tables:
     show_metadata('//lakehouse/default/Tables/' + table,table)
 
The ALTER TABLE statement changes the TBLPROPERTIES, for future writes on the table. Tables with the metadata defined, don't have the TBLPROPERTIES defined. In fact, the TBLPROPERTIES is not default. Tables with the TBLPROPERTIES defined may not have the metadata defined. 

I tested the ALTER TABLE and executed and OPTIMIZE after it. The table got the TBLPROPERTY for VORDER, but not the metadata for VORDER, making this very confuse.

Kind Regards,
 
Dennes



Kind Regards,

 

Dennes

 

data engineering

2 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi DennesTorres ,

    Thanks for using the Fabric community and reporting this . 

    I have reached to the internal for help on this. I will update you once I hear from them.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi DennesTorres,

      Thanks for reporting this and we can observe it as an issue. I have reached the internal Team regarding this. However, the Team is working on it and going to be fixed soon. I will inform if there is any update.

      Thanks for understanding.