Nested notebooks calls: Cannot identify Lakehouse files from child notebook

[prathijp]

I have notebook named NB_Orchestrate with the below call to another another notebook.

 mssparkutils.notebook.run(NB_xmlToParquet,300,"xml_data_pathRef":dataRef,"xml_struct":dataParseCodeRef,"saveParsedFile":saveParsedFile})

The child note book NB_xmlToParquet has the below code. When I execute the child note separately I am able to read the xml file and create the parquet file. But executed from the parent notebook it is throwing the below error. I tried with both relative filepath and abfs filepath

FileNotFoundError: [Errno 2] No such file or directory: ''You can check driver log or snapshot for detailed error info! See how to check logs: https://go.microsoft.com/fwlink/?linkid=2157243 .

  tree = ET.parse(xml_data_path)

  root = tree.getroot()

  # Extract data from XML

  data = {

      "owner": root.find("owner").text,

      "type": root.find("type").text,

      "values": [(value.attrib["date"], float(value.text)) for value in root.findall("series/value")]

  }

  # Create a DataFrame from the extracted data

  df = pd.DataFrame(data["values"], columns=["date", "value"])

  df["owner"] = data["owner"]

  df["type"] = data["type"]

  df["INS_DT"] = datetime.now()

  os.makedirs(os.path.dirname(saveParsedOuput), exist_ok=True)

  # Save the DataFrame to Parquet format

  df.to_parquet(saveParsedOuput, index=False)

[prathijp · Accepted solution]

Hi All,

Thank you for the suggestions. I did more exploration based on all the recommendations and got this worked.

The changes made

1. Made use of notebookutils to do nested notebook calls and lakehouse file operations.

2. File path was changed to  Relative path for Spark starting from File/ in all notebookutils methods.

3. Changed file related operations with spark commands like

  spark.read.format("xml").option("rowTag","data").load(filepath)

  df.write.format("parquet").load(save file) 

[prathijp]

Thank you OungiNeils

I have made the below changes

if notebookutils.fs.exists(xml_data_path):

  tree = ET.parse(xml_data_path)

  root = tree.getroot()

notebookutils.notebook.run(NB_xmlToParquet,300,"xml_data_path":dataRef,"xml_struct":dataParseCodeRef,"saveParsedFile":saveParsedFile})

When I print the variable I can see the filepath. However the notebook execution is throwing the below error: 

Py4JJavaError: An error occurred while calling o6443.throwExceptionIfHave. : com.microsoft.spark.notebook.msutils.NotebookExecutionException: Can not create a Path from an empty string ---------------------------------------------------------------------------IllegalArgumentException Traceback (most recent call last)Cell In[6], line 8

[v-veshwara-msft]

Hi @prathijp ,
Thanks for using Microsoft Fabric Community,

Thanks for the update.

Just a quick clarification on the usage of notebookutils.notebook.run(): when passing parameters, please make sure to provide them as a single dictionary object. The syntax you're currently using is interpreted incorrectly and can result in empty values being passed to the child notebook, which leads to the IllegalArgumentException you're seeing.

Corrected syntax:

notebookutils.notebook.run(
NB_xmlToParquet,
300,
{
"xml_data_path": dataRef,
"xml_struct": dataParseCodeRef,
"saveParsedFile": saveParsedFile
}
)

Also, ensure that you're reading the parameters in the child notebook using:

xml_data_path = notebookutils.notebook.get_context().notebook_param("xml_data_path")

This should help ensure the file path is passed and recognized correctly during execution.

Let us know if you’re still seeing issues after updating the syntax.

Also thanks to @ObungiNiels for your valuable response.

Hope this helps. Please reach out for further assistance.
If this post helps, then please consider to Accept as the solution to help the other members find it more quickly and a kudos would be appreciated.

Thank you.

[prathijp]

Thank you for the reply.

I tried using the below to retrieve the parameters. 

xml_data_pathNB = notebookutils.notebook.get_context().notebook_param("xml_data_path") 

However the method doesnot seem to be available - 

3 from datetime import datetime 4 import os ----> 6 xml_data_pathNB = notebookutils.notebook.get_context().notebook_param("xml_data_path") 7 removeFolderHeader = "" 8 xml_data_pathRef = "file:" + xml_data_pathNB.replace(removeFolderHeader,"") AttributeError: module 'notebookutils.notebook' has no attribute 'get_context'

But by removing the empty variable assignment I am able to get the parameters in the child notebook. However the file doesnot seem to be accessible.

xml_data_pathRef:   Files/Landing/CTC_SM.xml

[Errno 2] No such file or directory: 'Files/Landing/CTC_SM.xml' ---------------------------------------------------------------------------FileNotFoundError Traceback (most recent call last)Cell In[5], line 13 10 saveParsedFile = saveParquetFile.replace(removeFolderHeader,"") 12 if xml_struct == "xml_struct_01": ---> 13 xml_struct_01(xml_data_pathRef,saveParsedFile) Cell In[4], line 11, in xml_struct_01(xml_data_pathRef, saveParsedFile) 7 notebookutils.fs.head(xml_data_pathRef, 3000) 8 # Read the XML file content ---> 11 with open(xml_data_pathRef, "r", encoding="utf-8") as file: 12 xml_data = file.read() 14 # Parse XML 15 #root = ET.fromstring(xml_data)

Again if I use the below code I am able to view the content of the file.

notebookutils.fs.head(xml_data_pathRef, 3000)

[ObungiNiels]

Hi @prathijp , 

my guess would be that the functions from os library used for interacting with the file system in Fabric do not work when running the notebook from a separate one. 

I'd advice to use the notebookutils library instead of os to achieve the same. Here's the documentation . You will see that all functions you need are available in the library as well, for instance notebookutils.fs.mkdirs instead of  os.makedirs.

Addtionally, I'd use df.write.format("parquet").load(file_destination) instead. However, the error seems to occur with the directory, therefore the above would be the first thing I'd try to amend. 

Hope this helps! 🙂