Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Shape the future of the Fabric Community! Your insights matter. That’s why we created a quick survey to learn about your experience finding answers to technical questions. Take survey.

Reply
myon
Frequent Visitor

Iceberg, hudi for Fabric?

As a business user I love the seemlessness of fabric and I realize its potential. User experience is a big deal, and it seems really polished in Fabric. I started digging and asking our architects about our current mesh journey.

To my dissapointment it seems that Fabric is locked into delta lake extension of parquet? Is this going to be extended to support Iceberg/Hudi and be cloud store agnostic?

This is the answer I got from our architect:

Atleast from the data mesh perspective, we are trying to be technology agnostic. Every year, there will always be something new and something trend setting. Datamesh needs to adopt it or incorporate it and continue as an operating model which is not stuck on the tech.

1 ACCEPTED SOLUTION
cmaneu
Microsoft Employee
Microsoft Employee

Delta Lake format also supports ACID, and time travel can be implemented as well.

I haven't tested it myself, but I'm pretty sure you can read iceberg/parquet files stored into your S3 from OneLake with a Fabric Notebook. What you won't be able to do is to mount it as a Table in the Fabric world. But you can imagine having a bronze layer stored in S3 as Iceberg/parquet, and your upper layers stored in Fabric/Onelake as Delta.

 

About your question regarding Shortcuts, you can make a shortcut to an S3 Bucket either at the root of the bucket, or to a specific folder. The only limitation if you don't have delta-parquet file in you rS3 is that you won't be able to make that shortcut on the tables folder of your Fabric Lakehouse.

 

Here is a poster about OneLake shortcuts (from https://aka.ms/fabric-notes)

cmaneu_0-1685353994023.png

 

View solution in original post

5 REPLIES 5
R1k91
Solution Supplier
Solution Supplier

it's been official announced support for iceberg in Fabric later this year

https://www.snowflake.com/blog/microsoft-partnership-enhancing-interoperability/?lang=it


--
Riccardo Perico
BI & Power BI Engineer @ Lucient Italia

Blog | GitHub

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
cmaneu
Microsoft Employee
Microsoft Employee

Delta Lake format also supports ACID, and time travel can be implemented as well.

I haven't tested it myself, but I'm pretty sure you can read iceberg/parquet files stored into your S3 from OneLake with a Fabric Notebook. What you won't be able to do is to mount it as a Table in the Fabric world. But you can imagine having a bronze layer stored in S3 as Iceberg/parquet, and your upper layers stored in Fabric/Onelake as Delta.

 

About your question regarding Shortcuts, you can make a shortcut to an S3 Bucket either at the root of the bucket, or to a specific folder. The only limitation if you don't have delta-parquet file in you rS3 is that you won't be able to make that shortcut on the tables folder of your Fabric Lakehouse.

 

Here is a poster about OneLake shortcuts (from https://aka.ms/fabric-notes)

cmaneu_0-1685353994023.png

 

myon
Frequent Visitor

Thanks, this is good information.

cmaneu
Microsoft Employee
Microsoft Employee

Hello @myon,

Thanks for starting the discussion. 
First, when you said "locked", we need to clarify a few things: 
- The parquet format is open, as the delta format is open too Home | Delta Lake

- You can read and write other type of file formats within Fabric. But yes, all the storage for Fabric engines is written in the delta/parquet format
- You can integrate S3 buckets and soon Google Cloud Storage into OneLake (Fabric Storage) through shortcuts.
- You can implement a data mesh architecture with Fabric, based on an open lake (accessible APIs and open file format).

- From what I understand from Iceberg

That being said, I woud like to understand more of your interest on the Iceberg format, and why choosing it over the parquet format (which are both open formats from Apache, and are note tailored for the same usage). Also, You can submit that ask to the Fabric Ideas section of this site, and get people to vote on your idea :).

myon
Frequent Visitor

Iceberg is a direct competitor to Delta Lake in my understanding. They both sit on top of parquet files (or in the case of Iceberg they can be other columnar store files, like ORC) and give ACID, timetravel etc etc. I see the potential in Fabric and but I am wondering if it is compatible with our S3 + Iceberg+parquet setup.

 

A genuine question I have right now is, does OneLake need only the parquet files in S3 to make shortcuts or does it excplicitly need delta lake to be present in the S3 instance?

Helpful resources

Announcements
November Carousel

Fabric Community Update - November 2024

Find out what's new and trending in the Fabric Community.

Dec Fabric Community Survey

We want your feedback!

Your insights matter. That’s why we created a quick survey to learn about your experience finding answers to technical questions.

November Update

Fabric Monthly Update - November 2024

Check out the November 2024 Fabric update to learn about new features.

Live Sessions with Fabric DB

Be one of the first to start using Fabric Databases

Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.