Version Controlled Lakehouse
Version Controlled Lakehouse
Version Controlled Lakehouse is a Microsoft Fabric extensibility workload that brings Git-like version control semantics to data files stored in OneLake, directly inside the Fabric experience.
It enables users to explore, query, snapshot, and version CSV and Parquet files in a Lakehouse using familiar concepts such as repositories, branches, commits, and commit history.
Repository: https://github.com/ugljesanovak/fabric-lakehouse-versioning
Architecture: https://github.com/ugljesanovak/fabric-lakehouse-versioning/blob/master/docs/items/LakehouseGitFS/architecture.md
Demo: https://www.youtube.com/watch?v=T8huw007CPw
What Is It?
Version Controlled Lakehouse is a browser-first, metadata-driven versioning system for lakehouse files.
It allows users to treat a set of CSV and Parquet files in OneLake as a versioned repository, enabling:
- Branching and experimentation
- Immutable snapshots of data states
- Commit history and time-travel
- SQL-based exploration of data at any point in time
The design is inspired by Git workflows and data versioning systems such as lakeFS, while leveraging DuckDB WASM to provide interactive SQL querying directly in the browser.
All logic runs in the client, and all persisted state is stored using Fabric item metadata and OneLake storage.
What Problem Does It Solve?
Microsoft Fabric provides Git integration for certain workspace and item metadata, but it does not natively support:
- Versioning of lakehouse files
- Commit-based snapshots of data
- Branch-based experimentation on data
- Visual history or time-travel for files
Version Controlled Lakehouse addresses this gap by introducing a Git-like mental model for lakehouse data, allowing users to:
- Capture reproducible states of data
- Explore historical versions
- Safely experiment on branches
- Track how data evolved over time
This is particularly valuable for data engineering, analytics exploration, and reproducibility scenarios.
Who Is It For?
This workload is designed for:
- Data engineers experimenting with datasets
- Analytics engineers validating transformations
- Architects exploring data versioning patterns in Fabric
- Teams that want Git-style workflows applied to data files
It is also intended as a reference extensibility implementation, demonstrating how advanced data lifecycle concepts can be built on top of Fabric using the Extensibility Toolkit.
What Is Included in the MVP?
The current implementation focuses on a frontend-only MVP, with the following capabilities:
Repository and Branch Management
- Create and manage repositories
- Create, switch, and delete branches
- Track branch HEAD and commit ancestry
- Navigate repository → branch → file structure
File Exploration and SQL Querying
- Embedded DuckDB WASM SQL engine
- Query CSV and Parquet files using SQL
- Load multiple files for joins and analysis
- Query execution fully in the browser
Git-Like Versioning Model
- Commit model with parent relationships (DAG)
- Full repository snapshot captured per commit
- Support for “Save As” and “Overwrite” commit flows
- Commit history visualization
Each commit stores a snapshot of files under:
/Files/.gitfs/{item_id}/Data/{commit_id}
Metadata describing repositories, branches, commits, and files is stored in the Fabric item definition.
Storage and Persistence
- Files persisted in OneLake
- Metadata persisted via Fabric item definition
- No backend services required
How It Is Implemented (Architecture Summary)
Version Controlled Lakehouse follows a three-layer architecture:
UI Layer
- React with Fluent UI v9
- ItemEditor-based layout
- Repository explorer, SQL panel, commit graph
Execution Layer
- DuckDB WASM running in the browser
- Isolated execution per session
Persistence Layer
- Fabric item metadata for repository state
- OneLake file storage for commit snapshots
This architecture enables rapid iteration, simple deployment, and a clear separation between metadata and data storage.
Planned Future Enhancements
The following capabilities are explicitly out of scope for the MVP and planned as future enhancements:
- Git-style merge support with conflict resolution
- Diff visualization between commits or branches
- Backend metadata service for multi-user concurrency
- Zero-copy storage using OneLake shortcuts
- Storage deduplication via content hashing
- Integration with Fabric catalog, lineage, and governance services
- Policy-based controls for commits and branches
These enhancements would evolve the project from a frontend-driven prototype into a more production-ready data versioning system.
Why It Matters
Version Controlled Lakehouse demonstrates how data can be treated like code inside Microsoft Fabric, using familiar Git workflows applied to lakehouse files. It showcases the power of Fabric extensibility and highlights how advanced data lifecycle management patterns can be implemented with minimal infrastructure.