Forum Discussion
Helper ISave a pyspark dataframe in a table in warehouse using notebook
I am using notebook and have a pyspark dataframe , please guide me in saving the same as a overwrite mode table in warehouse inside a custom schema.
Arnab0111 Would suggest if you can navigate to data enginnering space from bottom left and go to Use sample section ,you will see the data engeering starter kit , it has many examples from writing t& transformation which will guide you how you can save table as default in Table section in a Lakehouse and data prep from data engineering point of view , which will save the table in default catalog of lakehouse
After that Open the Data warehouse and add your Lakehouse sql end point
Query the DBO Schema table from lakehouse and save as view in to your Custom Schma of your data warehouse
3 Replies
- puneetvijwani
Resolver IV
Arnab0111 As of now its not possible to save Pysparkdataframe to schema other than dbo ( if you choose to save them in Tables section for lakehouse)
You can create a viw from your lakehouse to custom schema or use sql endpoint of lakehose and create view in your datawarehouse under Custom Schema
Ps: If my response helped, kindly select it as the solution. Your kudos are greatly appreciated!
Arnab0111Can you please guide me in saving to default dbo schema only
- puneetvijwani
Arnab0111 Would suggest if you can navigate to data enginnering space from bottom left and go to Use sample section ,you will see the data engeering starter kit , it has many examples from writing t& transformation which will guide you how you can save table as default in Table section in a Lakehouse and data prep from data engineering point of view , which will save the table in default catalog of lakehouse
After that Open the Data warehouse and add your Lakehouse sql end point
Query the DBO Schema table from lakehouse and save as view in to your Custom Schma of your data warehouse