Forum Discussion

kely's avatar
kely
Frequent Visitor
1 year ago
Solved

cannot write to lakehouse with schema enabled 3 part naming

I am using a pyspark notebook, reading data from one lakehouse (not schema enabled) to another lakehouse (schema enabled) in the same workspace... however:   it ONLY works if I use: df.write.format...
data engineering
lakehouse
notebook
  • Anonymous's avatar
    Anonymous
    1 year ago

    Hi kely 

     

    My previous reply is based on my testing results, without any documentation supporting. So I'm not sure whether it's expected behavior, or it's a limitation, or it might be improved in the future. 

     

    If your notebook is currently using a 'bronze' lakehouse as the default lakehouse, using saveAsTable with 'goldlakehouse.schema.table' is probably not valid. Usually switching the default lakehouse of a running notebook will need to terminate the session first. If I understand it correctly, one session can have only one default lakehouse. So at least one lakehouse needs to use the ABFS path in the notebook. 

     

    As of now, I recommend using ABFS paths and save function to write the dataframe in your case. This is more reliable. 

     

    Best Regards,
    Jing