Apache Hudi and OneLake

[vitaly]

Has anyone been able to successfully read or write Apache Hudi files via Fabric Notebooks? If so, would you mind please sharing your approach?

We've tried a range of code approaches and seem to be running into a compatibility issue between the Hudi package and OneLake.

[v-kpoloju-msft]

Hi @vitaly,

Thank you for reaching out to the community and sharing the details of your scenario.

Currently, Apache Hudi is not officially supported in Microsoft Fabric Notebooks, particularly when interacting with One Lake. While Fabric Notebooks are powered by Apache Spark, the current managed runtime does not allow importing external JAR packages (like Hudi bundles), and One Lake does not expose a file system interface compatible with Hudi's timeline and metadata operations.

If your primary goal is to perform upserts, incremental processing, or ACID-compliant writes, I recommend using Delta Lake instead, which is natively supported in Microsoft Fabric and One Lake. Here is the one example shown below. Please go through them.

df.write.format("delta").mode("overwrite").save("Tables/my_delta_table")
df = spark.read.format("delta").load("Tables/my_delta_table")

You can find the list of supported file formats in One Lake here:

 One Lake indexer (preview) - Azure AI Search | Microsoft Learn

While it is technically possible to use Hudi in Apache Spark by including relevant JARs and configuring write options, Microsoft Fabric currently does not support external JAR imports or Spark reconfiguration using “%%configure.”
Apache Spark runtime in Fabric - Microsoft Fabric | Microsoft Learn

If your use case absolutely requires Hudi, you may consider:

• Running your Hudi workload in a Databricks or Synapse Spark environment
• Writing to an ADLS Gen2 container, then accessing it in Fabric through shortcuts or ingestion pipelines.

If your scenario requires Apache Hudi and full control over Spark JARs, you may consider using Azure HDInsight or Azure Databricks with One Lake integration:

Use OneLake with Azure HDInsight

Use OneLake with Azure Databricks

These environments support advanced file formats like Apache Hudi, and still allow data exchange with One Lake, even though they run outside the Microsoft Fabric runtime.

Hope this helps clarify things and let me know what you find after giving these steps a try happy to help you investigate this further.

Thank you for using the Microsoft Fabric Community Forum.

[vitaly]

Sorry, what's your source for the statment "Microsoft Fabric currently does not support external JAR imports or Spark reconfiguration using “%%configure”?

According to Manage Apache Spark libraries in Microsoft Fabric it's officially stated as possible. And it seems to work well from my experiments.

For our use case we specifically need Hudi, not Delta Lake. And the compatibility issue seems be specific to OneLake storage, not Fabric's Spark execution runtime.

[v-kpoloju-msft]

Hi @vitaly, 

Thank you for the clarification and you are right to point out the official documentation regarding custom library support in Microsoft Fabric.

You are referring to this article: Manage Libraries in Microsoft Fabric Notebooks

You are correct: as per the latest Fabric updates, custom libraries can now be managed at the workspace and session level, including via %pip and %mssparkutils.library.installPyPI. This indeed enables more flexibility when working with packages like Apache Hudi in Fabric Notebooks. To work arounds this issue:

Option 1: Use ADLS Gen2 via Shortcuts

• Write your Apache Hudi data to a separate ADLS Gen2 location (not under One Lake).
• Create a One Lake shortcut to that Hudi dataset (if read access from Fabric visuals is needed). Unify data sources with One Lake shortcuts - Microsoft Fabric | Microsoft Learn

Option 2: Run Full Hudi Workload Externally (HDInsight or Databricks)

• Execute the Hudi job in Azure HDInsight or Azure Databricks, where the file system access and Spark tuning options are unrestricted.
• Store results in ADLS Gen2 and use a shortcut into Fabric to make it accessible from downstream Fabric artifacts.
  Integrate One Lake with Azure Databricks - Microsoft Fabric | Microsoft Learn
  Integrate One Lake with Azure HDInsight - Microsoft Fabric | Microsoft Learn

Hope this helps clarify things and let me know what you find after giving these steps a try happy to help you investigate this further.

Thanks again for your insights and for contributing to the community discussion.

[vitaly]

Sorry, is this an AI-generated response? I am looking for expert opinion and community insight rather than GenAI.

[v-kpoloju-msft]

Hi @vitaly,
Apologize for the delayed response.

Store your Hudi data in ADLS Gen2 (outside One Lake). Use Apache XTable (Incubating) in Fabric to translate the Hudi dataset into Delta Lake format.
https://xtable.apache.org/docs/fabric/

Access the translated Delta dataset via standard Fabric engines (Spark, T-SQL, Power BI), leveraging One Lake or shortcuts. This ensures full compatibility and performance.
https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-and-delta-tables

Hope this helps. If you have any doubts regarding this, please feel free to ask here. We will be happy to help.

Thank you for using the Microsoft Community Forum.

[v-kpoloju-msft]

Hi @vitaly,

Just checking in to see if the issue has been resolved on your end. If the earlier suggestions helped, that’s great to hear! And if you’re still facing challenges, feel free to share more details happy to assist further.

Thank you.

[v-kpoloju-msft]

Hi @vitaly,

Hope you had a chance to try out the solution shared earlier. Let us know if anything needs further clarification or if there's an update from your side always here to help.

Thank you.

[vitaly]

@v-kpoloju-msft 

The specific requirement again is to access Hudi files hosted in OneLake from Fabric Notebooks. Native OneLake storage, not shortcuts. Fabric Notebooks with Spark for the runtime, not any other runtimes such as T-SQL, etc. Hudi format, not Delta Lake, Parquet or other formats.

I've reviewed the links and explanation you provided and do not see anything that addresses this specific requirement. Please let me know if I missed it.

[v-kpoloju-msft]

Hi @vitaly,

Since you are facing issues while integrating lakehouse with apachi hudi, please raise your issue here: Issues - Microsoft Fabric Community

Thank you.