Forum Discussion

sjpark's avatar
sjpark
Frequent Visitor
2 years ago
Solved

V-Order & Z-Order

Hi, i'm using Fabric.

 

And i got a simple question.

 

What is the difference between V-Order and Z-Order?

 

I know V-Order is a write time optimization to the parquet file format that enables lightning-fast reads under the Microsoft Fabric compute engines, such as Power BI, SQL, Spark, and others.(as MS Docs said)

 

But i can't find any Docs of Z-Order, and even i can't understood well after reading V-Order & Z-Order disc.

 

Please help this newbie, i will really appricate all your comments.

 

Thanks a lot.

Park.

data engineering
data warehouse
  • Anonymous's avatar
    Anonymous
    2 years ago

    Hi sjpark 
    Thanks for using Fabric Community.

     

    1. V-Order:

      V-Order is a write-time optimization specifically designed for the Parquet file format within the Microsoft Fabric ecosystem. Its primary goal is to enhance read performance under various compute engines, including Power BI, SQL, and Spark.
      • Key features of V-Order:
        • Sorting: V-Order applies special sorting techniques to the Parquet files.
        • Row Group Distribution: It optimizes row group distribution.
        • Dictionary Encoding: Efficient dictionary encoding is used.
        • Compression: V-Order achieves better compression, leading to reduced storage costs.
      • Benefits:
        • Lightning-Fast Reads: Power BI and SQL engines leverage Microsoft Verti-Scan technology and V-Ordered Parquet files, resulting in in-memory-like data access times.
        • Performance Boost: Even non-Verti-Scan compute engines (like Spark) benefit from V-Ordered files, with an average of 10% faster read times (up to 50% in some scenarios).
        • Cost Efficiency: V-Order reduces network, disk, and CPU resources during reads.
      • Compatibility:
        • V-Order is 100% open-source Parquet format compliant, meaning all Parquet engines can read it as regular Parquet files.
        • It works seamlessly with Delta tables and features like Z-Order, compaction, vacuum, and time travel.

    2. Z-Order:

      Z-Order is another optimization technique, but it’s not specific to Fabric; it’s widely used in data lakes and analytics platforms. Z-Order aims to improve query performance by co-locating related information in the same set of files.
      • How It Works:
        • Z-Order organizes data based on one or more columns (usually categorical or frequently filtered columns).
        • Rows with similar values in the specified columns are stored together.
        • This co-locality reduces the amount of data that needs to be read during queries.
      • Benefits:
        • Data Skipping: By avoiding unnecessary reads, Z-Order significantly improves query efficiency.
        • Compatible with Delta Lake: Z-Order works seamlessly with Delta Lake.

    Key Differences:

    • Timing: V-Order happens during write time, Z-Order during read time (or table optimization).
    • Purpose: V-Order focuses on compression and general read performance, Z-Order on co-locating data for specific queries.
    • Compatibility: V-Order is universally compatible, Z-Order might require specific tools like Delta Lake.


    For more information please refer to these links:
    https://www.linkedin.com/posts/lucazanna_data-microsoft-fabric-activity-7068093014677540864-hNm0/?originalSubdomain=hk
    https://docs.delta.io/latest/optimizations-oss.html#language-sql
    https://learn.microsoft.com/en-us/fabric/data-engineering/delta-optimization-and-v-order?tabs=sparksql
    https://www.dremio.com/blog/how-z-ordering-in-apache-iceberg-helps-improve-performance/
    Data skipping for Delta Lake - Azure Databricks | Microsoft Learn

    Hope this helps. Please let me know if you have any further questions. Glad to help.

6 Replies