topic Re: dataframe to table using python in notebook in Data Engineering

dataframe to table using python in notebook

munindra — Tue, 31 Dec 2024 03:41:45 GMT

i am working with vector embedding inside the notebook using python while i store the result into the dataframe i am unable to create table inside the lakehouse out of it when i am using python. but i need to work in python do this process and not with with pyspark and this is a intermediate step inisde the notebook so i can't use pipeline to convert csv to table .so anyone help regarding this issue.

Re: dataframe to table using python in notebook

Anonymous — Tue, 31 Dec 2024 08:42:29 GMT

Hi @munindra,

Thanks for reaching out to the Microsoft fabric community forum.

To create a table in the lakehouse directly from a DataFrame using Python, you can utilize the Pandas API to write the DataFrame to a format compatible with the lakehouse, such as Parquet or Delta. Below is a step-by-step guide:

Before adding any code to your notebook, please ensure that you manually select Python in the notebook to execute the process, as shown in the picture below:

Ensure you have the necessary libraries installed. You can install them using pip if they are not already installed use the below code to install:

!pip install pandas pyarrow

Please use Pandas to save your DataFrame as a Parquet file. Parquet is a columnar storage format that optimizes both storage efficiency and query performance. Us

import pandas as pd

df.to_parquet('/lakehouse/default/Files/your_table.parquet')# assume dataframe as df.

Use the lakehouse API to load the Parquet file into a table. Also, this step might vary depending on the specific lakehouse platform you are using and for your reference below is the function to load DataFrame into the lakehouse table.

load_to_lakehouse_table(df, table_name='your_table')

For further information regarding pandas, please refer to the official Microsoft document below.

Read and write data with Pandas - Microsoft Fabric | Microsoft Learn

I hope my suggestions give you good ideas, if you need any further assistance, please feel free to reach out.

If this post helps, then please give us 'Kudos' and consider 'Accept it as a solution' to help the other members find it more quickly.

Thank you.

Re: dataframe to table using python in notebook

Anonymous — Fri, 03 Jan 2025 02:13:00 GMT

Hi @munindra,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

Re: dataframe to table using python in notebook

Anonymous — Mon, 06 Jan 2025 08:03:41 GMT

Hi @munindra,

I wanted to follow up on our previous suggestions regarding dataframe to table using python in notebook. We would love to hear back from you to ensure we can assist you further.

If my response has addressed your query, please accept it as a solution and give a ‘Kudos’ so other members can easily find it. Please let us know if there’s anything else we can do to help.

Thank you.

Re: dataframe to table using python in notebook

munindra — Mon, 06 Jan 2025 10:10:33 GMT

sorry for the late respose
although saveasTable is working it was creating the table only when i mentioned table name without location and i am unable to find at what loaction the table was being created and if use the partial path or complete path i was returning error

Re: dataframe to table using python in notebook

Anonymous — Fri, 10 Jan 2025 09:43:42 GMT

Hi @munindra,

We really apologize for delay in response, and thanks for bringing your concerns to us as you mentioned you are unable to find the location of the table as a result this will be stored at the DEFAULT lake house please once have a look at that default location you can find the table over there and also you have raised an one more concern that if you are giving the location or path you are not able to create a table and it is throwing the error to fix the issue that you are facing please refer the below document mention and for your convenience I have also pasted the screenshot please have a look on it.

https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-notebook-load-data#load-data-with-pandas-api

Additionally, to Use Python experience on Notebook please refer the below provided document:

Use Python experience on Notebook - Microsoft Fabric | Microsoft Learn

If this post helps, then please give us 'Kudos' and consider 'Accept it as a solution' to help the other members find it more quickly.

Thank you.

Re: dataframe to table using python in notebook

frithjof_v — Sun, 12 Jan 2025 10:56:56 GMT

In the vanilla Python Notebook experience (preview), you can find code snippets for writing to delta table without using Spark:

https://learn.microsoft.com/en-us/fabric/data-engineering/using-python-experience-on-notebook

https://learn.microsoft.com/en-us/fabric/data-engineering/using-python-experience-on-notebook#browse-code-snippets

Here are also some code examples:

https://www.reddit.com/r/MicrosoftFabric/s/CtyqVHSDKq

I think you will need to use the abfss path of the Lakehouse in order to do this.

Re: dataframe to table using python in notebook

Anonymous — Tue, 14 Jan 2025 04:46:43 GMT

Hello @munindra ,

I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.

Thank you.

Re: dataframe to table using python in notebook

Anonymous — Fri, 17 Jan 2025 14:02:58 GMT

Hi @munindra ,

May I ask if you have resolved this issue? If so, please mark the helpful reply and accept it as the solution. This will be helpful for other community members who have similar problems to solve it faster.

Thank you.