Coauthor: Iddo Avneri (Treeverse)
Use a live branch for up-to-date analytics or pin a shortcut to an immutable commit when reproducibility matters.
Data teams often need two seemingly conflicting things: immediate access to the latest data and the ability to return to the exact data behind an earlier analysis, report, or experiment.
Maintaining both typically means creating snapshots, copying datasets, or building additional pipelines. That adds operational overhead and makes it harder to know which version of the data was used.
Microsoft OneLake and lakeFS provide a simpler, complementary pattern for addressing this use case.
OneLake shortcuts allow Microsoft Fabric lakehouses to reference data in external storage through a unified namespace. Because lakeFS exposes versioned data through an S3-compatible gateway, Fabric can access lakeFS-managed datasets by using a OneLake S3-compatible shortcut.
The result: Fabric users can access either the current data on a lakeFS branch or an immutable version from a specific commit. This removes the need to create and maintain a separate pipeline for copying versioned data.
Figure: Access lakeFS data from Microsoft Fabric using shortcuts to either a live branch or a pinned commit.Live data and reproducible snapshots
A lakeFS repository organizes data using branches and immutable commits.
A branch, such as main, advances as new data versions are committed. A commit identifies a specific repository state and remains unchanged.
This makes it possible to create different OneLake shortcuts for different purposes:
- A branch-following shortcut gives Fabric users access to the current state of a dataset.
- A commit-pinned shortcut provides a stable snapshot for a report, experiment, audit, or other reproducible workload.
- A Delta table shortcut exposes a lakeFS-managed Delta table through the Fabric lakehouse and SQL analytics endpoint.
All three shortcuts access the same lakeFS repository, but each provides the data contract appropriate for its workload.
How the integration works
You can connect Fabric to the lakeFS S3-compatible endpoint with scoped access credentials, then configure the shortcut to reference the required repository version and path.
- The lakeFS repository is addressed as the S3-compatible bucket.
- A branch name, commit ID, or tag identifies the desired version.
- The remaining path identifies the files or Delta table to expose in Fabric.
For example, a Fabric lakehouse could use separate shortcuts for current data, a reproducible snapshot, and a Delta table:
- lakefs-current-data, pointing to a dataset on the main branch.
- lakefs-reproducible-snapshot, pointing to a specific lakeFS commit.
- lakefs-delta-table, pointing to a Delta table on main.
Files can be accessed through the lakehouse’s Files area. A Delta table shortcut created under Tables can be queried through the Fabric SQL analytics endpoint.
The data remains managed and versioned in lakeFS, while Fabric users access it through the environment and analytics experiences, they already use.
What we validated
We validated this pattern end to end using the official Microsoft Fabric REST APIs. Based on our testing, we arrived at the following conclusions:
- Fabric successfully connected to the lakeFS S3-compatible gateway.
- We created shortcuts to both a lakeFS branch and a specific lakeFS commit.
- After committing new data to main, the branch-following shortcut reflected the updated state.
- The commit-pinned shortcut continued to return the original, byte-identical data.
- A shortcut to a Delta table was recognized in the Fabric lakehouse and could be queried through the SQL analytics endpoint.
- In the validated configuration, Fabric could read data through the shortcuts but could not write changes back to lakeFS.
Where this pattern helps
Consider a team producing a recurring risk report.
The data engineering team publishes new data versions to the main branch in lakeFS. The report can use a branch-following OneLake shortcut to access the latest published data in Fabric.
When the report is finalized, the team can record (or create a shortcut to) the exact lakeFS commit used for that reporting period. Even after main advances, the pinned shortcut continues to expose the original data version.
The same pattern can support:
- Reproducing an earlier analysis or model result.
- Providing auditors with the exact data used for a report.
- Comparing the latest dataset with a previously approved version.
- Giving different Fabric workloads stable or continuously updated views of the same source data.
- Reducing the number of copied snapshots and custom synchronization pipelines teams need to maintain.
Getting started
To use this integration pattern:
- Make the lakeFS S3-compatible gateway accessible from Microsoft Fabric.
- Create an S3-compatible connection in Microsoft Fabric using the lakeFS endpoint and scoped access credentials.
- Create a shortcut to the appropriate lakeFS repository, reference, and path.
- Use a branch reference for continuously updated data or a commit reference for a fixed, reproducible snapshot.
- For Delta data, create the shortcut under Tables so the table can be queried through the Fabric SQL analytics endpoint.
Microsoft’s current S3-compatible shortcut documentation specifies access-key and secret-key authentication. Credentials should therefore be scoped and managed according to the organization’s security requirements.
Access current and reproducible versions of data in Fabric
OneLake shortcuts make externally managed data accessible across Microsoft Fabric without requiring a separate copy and staging workflow. lakeFS adds the ability to choose precisely which version Microsoft Fabric sees.
Together, OneLake shortcuts and lakeFS let teams analyze the latest published data while retaining immutable data versions for reports, experiments, and audits. Teams can support both current and reproducible analytics in Fabric without maintaining separate data-copy pipelines.
Learn more
Microsoft Employee