Forum Discussion

prathijp's avatar
prathijp
Icon for Helper I rankHelper I
1 year ago
Solved

Nested notebooks calls: Cannot identify Lakehouse files from child notebook

I have notebook named NB_Orchestrate with the below call to another another notebook.  mssparkutils.notebook.run(NB_xmlToParquet,300,"xml_data_pathRef":dataRef,"xml_struct":dataParseCodeRef,"savePa...
data engineering
lakehouse
notebook
  • prathijp's avatar
    prathijp
    1 year ago

    Hi All,

    Thank you for the suggestions. I did more exploration based on all the recommendations and got this worked.

    The changes made

    1. Made use of notebookutils to do nested notebook calls and lakehouse file operations.

    2. File path was changed to  Relative path for Spark starting from File/ in all notebookutils methods.

    3. Changed file related operations with spark commands like

      spark.read.format("xml").option("rowTag","data").load(filepath)

      df.write.format("parquet").load(save file) 

ObungiNiels's avatar
ObungiNiels
Icon for Resolver III rankResolver III
1 year ago

Hi prathijp , 

my guess would be that the functions from os library used for interacting with the file system in Fabric do not work when running the notebook from a separate one. 

I'd advice to use the notebookutils library instead of os to achieve the same. Here's the documentation . You will see that all functions you need are available in the library as well, for instance notebookutils.fs.mkdirs instead of  os.makedirs.

 

Addtionally, I'd use df.write.format("parquet").load(file_destination) instead. However, the error seems to occur with the directory, therefore the above would be the first thing I'd try to amend. 

 

Hope this helps! 🙂