Use Spark Job Definitions to load data into Fabric warehouse

[pavannarani]

Hello,

I have a requirement to do some transformations using spark job definitions which will be scheduled to process daily or weekly. Currently am loading the data from azure blob to warehouse tables using direct load. Later am connecting semantic models to  warehouse tables to perform transformations and creating measures and calculation groups using import mode.

So I am thinking to use spark job definitions by fetching data from warehouse tables and perform transformations in it and also thinking to create calculated column(created based on existing columns) transformations directly in this spark jobs. I am wondering can i do that is it possible i feel using existing columns from every table its possible. Can any one suggest is this approach is correct or should i use Semantic model only. The idea is to reduce the creation of calculated columns in model and do it in spark jobs. Did anyone went through this scenario want to hear the thoughts is it possible.

Also can i load data to warehouse tables directly from spark jobs or should i use only lakehouse as target for spark jobs. ???

[pavannarani]

HI @v-yaningy-msft 

Thanks for your response for the first question i will try to implement this using spark Jobs.

For the second response am not very clear. You are suggesting to use a connector or API to load data to fabric warehouse tables from spark jobs. I searched for warehouse APIs didnt found any suitable blog. Is there any blog which explains the fabric warehouse APIs and how they will allow us to write the data to warehouse tables using spark jobs. Can you give more details on it or any example if possible how to load data to warehouse table with spark jobs.

Regards

Pavan kumar

[v-yaningy-msft]

Hi, @pavannarani 

You can use Spark job definitions to perform transformations on your data, including the creation of calculated columns based on existing columns from every table. Spark provides a robust framework for data processing, allowing you to apply complex transformations and calculations directly within your Spark jobs. This approach can indeed help reduce the need for creating calculated columns in your semantic model, potentially simplifying your data model and improving performance.

Lakehouse tutorial - Prepare and transform lakehouse data - Microsoft Fabric | Microsoft Learn

Regarding your question about loading data directly to warehouse tables from Spark jobs, it is possible to load data into warehouse tables, but the approach might vary based on your specific warehouse setup and the capabilities it supports. Typically, you would use Spark to process and prepare your data, and then utilize a connector or API provided by your warehouse to load the data. This process can be automated within your Spark job, allowing for seamless data loading after transformations.

Options to get data into the Lakehouse - Microsoft Fabric | Microsoft Learn

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

How to get your questions answered quickly --  How to provide sample data in the Power BI Forum