Forum Discussion

prathijp's avatar
prathijp
Icon for Helper I rankHelper I
1 year ago
Solved

Nested notebooks calls: Cannot identify Lakehouse files from child notebook

I have notebook named NB_Orchestrate with the below call to another another notebook.  mssparkutils.notebook.run(NB_xmlToParquet,300,"xml_data_pathRef":dataRef,"xml_struct":dataParseCodeRef,"savePa...
data engineering
lakehouse
notebook
  • prathijp's avatar
    prathijp
    1 year ago

    Hi All,

    Thank you for the suggestions. I did more exploration based on all the recommendations and got this worked.

    The changes made

    1. Made use of notebookutils to do nested notebook calls and lakehouse file operations.

    2. File path was changed to  Relative path for Spark starting from File/ in all notebookutils methods.

    3. Changed file related operations with spark commands like

      spark.read.format("xml").option("rowTag","data").load(filepath)

      df.write.format("parquet").load(save file) 

prathijp's avatar
prathijp
Icon for Helper I rankHelper I
1 year ago

Hi All,

Thank you for the suggestions. I did more exploration based on all the recommendations and got this worked.

The changes made

1. Made use of notebookutils to do nested notebook calls and lakehouse file operations.

2. File path was changed to  Relative path for Spark starting from File/ in all notebookutils methods.

3. Changed file related operations with spark commands like

  spark.read.format("xml").option("rowTag","data").load(filepath)

  df.write.format("parquet").load(save file)