Forum Discussion
Using open source ETL tools for Fabric Lakehouse/DWH
Hi, is there crrently no way to use open source tools to load data into Fabric lakehouse?
For example if we want use Airbyte and Airflow, or you have to use only Data Factory to load and transform data.
Thanks in advance.
16 Replies
Hi Anonymous
Thanks for using Fabric Community.
There are several ways to load data into Fabric Lakehouse:
Data Factory: These services can be used to copy data from and to Microsoft Fabric Lakehouse and transform data in Microsoft Fabric Lakehouse. You can also use a pipeline to load sample data into Lakehouse at high performance copy and then transform the data by dataflow.
Apache Spark: You can use Apache Spark libraries in notebook code to connect to a data source directly, load data to a data frame, and then save it in a lakehouse.
While Airbyte can connect to various sources and transform data, it currently doesn't have native support for writing directly to Fabric Lakehouse. Appreciate if you could share the feedback on our feedback channel, which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.Feedback Link: https://ideas.fabric.microsoft.com/
I hope this information is helpful. Please let me know if you have any other questions.You didn't reply specifically to the question itself. Spark notebooks and ADF are inside Fabric.
Is there currently a way to use outside ETL tools to load data into Fabric DWH. Like Airflow or Airbyte, I am not asking about native connections, but is it even possible to load data into Fabric DWH with anything outside Fabric itself.
Thanks in advance.We can use Fivetran(If that's helpful) to ingest data to lakehouse but it's not open source like Airflow or Airbytes I'm afraid.
