Forum Discussion
dataframe to table using python in notebook
i am working with vector embedding inside the notebook using python while i store the result into the dataframe i am unable to create table inside the lakehouse out of it when i am using python. but i need to work in python do this process and not with with pyspark and this is a intermediate step inisde the notebook so i can't use pipeline to convert csv to table .so anyone help regarding this issue.
Hi munindra,
We really apologize for delay in response, and thanks for bringing your concerns to us as you mentioned you are unable to find the location of the table as a result this will be stored at the DEFAULT lake house please once have a look at that default location you can find the table over there and also you have raised an one more concern that if you are giving the location or path you are not able to create a table and it is throwing the error to fix the issue that you are facing please refer the below document mention and for your convenience I have also pasted the screenshot please have a look on it.
Additionally, to Use Python experience on Notebook please refer the below provided document:
Use Python experience on Notebook - Microsoft Fabric | Microsoft Learn
If this post helps, then please give us 'Kudos' and consider 'Accept it as a solution' to help the other members find it more quickly.
Thank you.
8 Replies
In the vanilla Python Notebook experience (preview), you can find code snippets for writing to delta table without using Spark:
https://learn.microsoft.com/en-us/fabric/data-engineering/using-python-experience-on-notebook
Here are also some code examples:
https://www.reddit.com/r/MicrosoftFabric/s/CtyqVHSDKq
I think you will need to use the abfss path of the Lakehouse in order to do this.
Hi munindra,
Thanks for reaching out to the Microsoft fabric community forum.
To create a table in the lakehouse directly from a DataFrame using Python, you can utilize the Pandas API to write the DataFrame to a format compatible with the lakehouse, such as Parquet or Delta. Below is a step-by-step guide:
- Before adding any code to your notebook, please ensure that you manually select Python in the notebook to execute the process, as shown in the picture below:
- Ensure you have the necessary libraries installed. You can install them using pip if they are not already installed use the below code to install:
!pip install pandas pyarrow
- Please use Pandas to save your DataFrame as a Parquet file. Parquet is a columnar storage format that optimizes both storage efficiency and query performance. Us
import pandas as pd
df.to_parquet('/lakehouse/default/Files/your_table.parquet')# assume dataframe as df.
- Use the lakehouse API to load the Parquet file into a table. Also, this step might vary depending on the specific lakehouse platform you are using and for your reference below is the function to load DataFrame into the lakehouse table.
load_to_lakehouse_table(df, table_name='your_table')
For further information regarding pandas, please refer to the official Microsoft document below.
Read and write data with Pandas - Microsoft Fabric | Microsoft Learn
I hope my suggestions give you good ideas, if you need any further assistance, please feel free to reach out.
If this post helps, then please give us 'Kudos' and consider 'Accept it as a solution' to help the other members find it more quickly.
Thank you.
Hi munindra,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.Hi munindra,
I wanted to follow up on our previous suggestions regarding dataframe to table using python in notebook. We would love to hear back from you to ensure we can assist you further.
If my response has addressed your query, please accept it as a solution and give a ‘Kudos’ so other members can easily find it. Please let us know if there’s anything else we can do to help.
Thank you.
sorry for the late respose
although saveasTable is working it was creating the table only when i mentioned table name without location and i am unable to find at what loaction the table was being created and if use the partial path or complete path i was returning errorHi munindra,
We really apologize for delay in response, and thanks for bringing your concerns to us as you mentioned you are unable to find the location of the table as a result this will be stored at the DEFAULT lake house please once have a look at that default location you can find the table over there and also you have raised an one more concern that if you are giving the location or path you are not able to create a table and it is throwing the error to fix the issue that you are facing please refer the below document mention and for your convenience I have also pasted the screenshot please have a look on it.
Additionally, to Use Python experience on Notebook please refer the below provided document:
Use Python experience on Notebook - Microsoft Fabric | Microsoft Learn
If this post helps, then please give us 'Kudos' and consider 'Accept it as a solution' to help the other members find it more quickly.
Thank you.
Hello munindra ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.
