topic Re: Load Parquet File into Fabric Datawarehouse in Data Warehouse

Load Parquet File into Fabric Datawarehouse

westf — Tue, 19 Nov 2024 20:53:10 GMT

Hello,

Im writting a Fabric notebook to load a Parquet file into Fabric Warehouse.

I want this isn Pysparks only as the notebook is fully written using the PySparks. Please advise me the fast & efficient way to load the data into Warehouse from lakehouse.

My request is, If table exist, i want to upsert, if not, i want to create a table and insert the data.

Please kindly share code

Re: Load Parquet File into Fabric Datawarehouse

NandanHegde — Wed, 20 Nov 2024 05:29:52 GMT

Below is the sample code that we use :

# Load the Parquet file into a Spark DataFrame

df = spark.read.parquet("path/to/your/parquet/file")

# Write the DataFrame to the Fabric Data Warehouse df.write.mode("overwrite").saveAsTable("your_table_name")

Now based on what I know, you can either append or overwrite data in table directly.

I am not sure w.r.t upsert, need to validate myself

Re: Load Parquet File into Fabric Datawarehouse

NandanHegde — Wed, 20 Nov 2024 05:31:20 GMT

Note: you can use Spark SQL for upsert
What I meant is I am not sure myself whether it is doable via pyspark

Re: Load Parquet File into Fabric Datawarehouse

Anonymous — Wed, 20 Nov 2024 05:35:01 GMT

Hi westf,

Perhaps you can take a look at the following link to use notebook load data to data warehouse.

Load data to MS Fabric Warehouse from notebook - Stack Overflow

Regards,

Xiaoxin Sheng