Forum Discussion

PriyaJha's avatar
PriyaJha
Helper I
1 year ago
Solved

How to create a Lakehouse table without manually mounting Lakehouse in notebook

Hi All,

 

I want to convert Lakehouse files to Lakehouse tables using Notebooks but without mounting Lakehouse in notebook.

I dont want to mount the Lakehouse, as i am trying to create a generic notebook which is capable of converting any file of any Lakehouse into table in that particular lakehouse.

 

Things which i tried-

I am able to read the file providing the absolute path and store data in df but while trying to saveAsTable i am getting following error since Lakehouse is not mounted:

 

df = spark.read.parquet("abfss://[email protected]/LakehouseName.Lakehouse/Files/POC/FileName.parquet")
df.count()
df.write.format("delta").mode("overwrite").saveAsTable("TableName")

 

 

 

I tried mounting the Lakehouse via code and then use saveAsTable, but it is failing with syntax error:

 

 

import os
import pandas as pd

workspaceID = ""
lakehouseID = ""
mount_name = "/temp_mnt"

base_path = f"abfss://{workspaceID}@onelake.dfs.fabric.microsoft.com/{lakehouseID}/"
mssparkutils.fs.mount(base_path, mount_name)

df = spark.read.parquet(base_path+"/Files/FileName.parquet")
df.count()

table_path = base_path+"Tables/TableName"

df.write.format("delta").mode("overwrite").saveAsTable(table_path)

 

 

 

I think, i am performing some coding error and not able to provide table path correctly.

 

Kindly suggest some ways to acheive this scenario.

Lakehouse
Notebook
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hi PriyaJha ,

     

    To achieve the goal without mounting Lakehouse, use the Save method instead of saveAsTable:

    df = spark.read.csv(“abfss://[email protected]/daisyTest2.Lakehouse/Files/ProductsTest.csv”, header=True, inferSchema=True)

df.count()

table_path = “abfss://[email protected]/daisyTest2.Lakehouse/Tables/ProductsTest”
df.write.format(“delta”).mode(“overwrite”).save(table_path)

     

    Replace the workspaceName, lakehouseName, and csv file in it with your own to use.

     

    You can see that it works fine.

     

    You can see the table being loaded after hitting refresh at the Tables in lakehouse.

     

    If you have any other questions please feel free to contact me.

     

    Best Regards,
    Yang
    Community Support Team

     

    If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!

1 Reply

  • Anonymous's avatar
    Anonymous
    Not applicable
    1 year ago

    Hi PriyaJha ,

     

    To achieve the goal without mounting Lakehouse, use the Save method instead of saveAsTable:

    df = spark.read.csv(“abfss://[email protected]/daisyTest2.Lakehouse/Files/ProductsTest.csv”, header=True, inferSchema=True)

df.count()

table_path = “abfss://[email protected]/daisyTest2.Lakehouse/Tables/ProductsTest”
df.write.format(“delta”).mode(“overwrite”).save(table_path)

     

    Replace the workspaceName, lakehouseName, and csv file in it with your own to use.

     

    You can see that it works fine.

     

    You can see the table being loaded after hitting refresh at the Tables in lakehouse.

     

    If you have any other questions please feel free to contact me.

     

    Best Regards,
    Yang
    Community Support Team

     

    If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
    If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!