topic Save a pyspark dataframe in a table in warehouse using notebook in Data Warehouse

Save a pyspark dataframe in a table in warehouse using notebook

Arnab0111 — Tue, 29 Aug 2023 04:25:26 GMT

I am using notebook and have a pyspark dataframe , please guide me in saving the same as a overwrite mode table in warehouse inside a custom schema.

Re: Save a pyspark dataframe in a table in warehouse using notebook

puneetvijwani — Tue, 29 Aug 2023 08:02:35 GMT

Arnab0111 As of now its not possible to save Pysparkdataframe to schema other than dbo ( if you choose to save them in Tables section for lakehouse)

You can create a viw from your lakehouse to custom schema or use sql endpoint of lakehose and create view in your datawarehouse under Custom Schema

Ps: If my response helped, kindly select it as the solution. Your kudos are greatly appreciated!

Re: Save a pyspark dataframe in a table in warehouse using notebook

Arnab0111 — Tue, 29 Aug 2023 09:19:38 GMT

Can you please guide me in saving to default dbo schema only

Re: Save a pyspark dataframe in a table in warehouse using notebook

puneetvijwani — Tue, 29 Aug 2023 13:31:36 GMT

Arnab0111 Would suggest if you can navigate to data enginnering space from bottom left and go to Use sample section ,you will see the data engeering starter kit , it has many examples from writing t& transformation which will guide you how you can save table as default in Table section in a Lakehouse and data prep from data engineering point of view , which will save the table in default catalog of lakehouse

After that Open the Data warehouse and add your Lakehouse sql end point

Query the DBO Schema table from lakehouse and save as view in to your Custom Schma of your data warehouse