Connecting and pinning Lakehouse to the Notebook

[anusha_2023]

I have a scheduled notebook running on daily basis which is connected with Lakehouse for reading input parquet files and writing output as Tables. These tables are connected with Power BI report and semantic model. Now my daily schduled script is failing because pinned Lakehouse to the Notebook  is getting disconnected and I need to add the lakehouse everyday manually and re-run the script. May I know workaround for this issue?

[v-cboorla-msft]

Hi @anusha_2023 

Glad that your query got resolved.

Please continue using Fabric Community for any help regarding your queries.

[HimanshuS-msft]

Hello @anusha_2023 
Thanks for using the Fabric community.
I did tried to repro the issue and as I understand , the issue here is to the notebooks scheduled runs are failing .

I tried this to read a file and write the content to a table , 

df = spark.read.option("multiline", "true").json("Files/JSON/test.json")

# df now is a Spark DataFrame containing JSON data from "Files/JSON/test.json".

df.write.mode("append").format("delta").saveAsTable("test_table")

display(df)

and then I scheduled it for every 10 mins . The run just went fine without any failure . 

I am sure that I am missing something here . What is the error you are getting ?

Thanks
HImanshu

[anusha_2023]

I tried with sample data by pinning the lakehouse and loading as table with a new notebook shedules are successful. 

I have issue with my daily scheduled notebooks.

This is the error: 

Py4JJavaError: An error occurred while calling o6337.parquet. : java.io.FileNotFoundException: Operation failed: "Not Found", 404, PUT, http://onelake.dfs.fabric.microsoft.com/b04579d1-31c1-4194-a912-9f15db327234/6aa4d65e-dc1f-4212-8c1f... ArtifactNotFound, "Artifact '6aa4d65e-dc1f-4212-8c1f-1cafc20c20d5' is not found in workspace 'b04579d1-31c1-4194-a912-9f15db327234'."

Caused by: Operation failed: "Not Found", 404, PUT, http://onelake.dfs.fabric.microsoft.com/b04579d1-31c1-4194-a912-9f15db327234/6aa4d65e-dc1f-4212-8c1f... ArtifactNotFound, "Artifact '6aa4d65e-dc1f-4212-8c1f-1cafc20c20d5' is not found in workspace 'b04579d1-31c1-4194-a912-9f15db327234'." at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:231) at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:191) 

As a work aroud If I gave absolute path then I could able to write as parquet file. But saveAsTable is not working with absolute path as below.

df_bookings.write.mode("overwrite").format("delta").saveAsTable("<absolute-path>/Tables/bookingsdailyupdate")

When I am using save option instead I could not able to save to the default Tables area of the Lakehouse. The system is suggesting me to move to files folderand they are saving as delta files in the Files folder of the lakehouse.

df_bookings.write.mode("overwrite").format("delta").save("<absolute-path>/Tables/bookingsdailyupdate") 

[govindarajan_d · Accepted solution]

If you clone the notebook and then pin the lakehouse to the cloned notebook, does the same happen?

[anusha_2023]

If I made another copy and schedule the new notebook its worked at first time. Thanks for the tip. But second time schedule is successful but script did not ran. As below showing the monitor hub first run is successful at 2:23AM showing in the first screen shot and next runs are showing succeeded but check the setting details in the second screenshot below.

Notebook is not really running the script. Does that mean am I am ran out of ran capacity or what would be the reason behind?

[govindarajan_d]

How are you saying the notebook did not execute? The data does not get written to table?

If you go to recent runs, you will be able to individually go into each run's status and you can click on item snapshots to see how the notebook has run. Please check that and see if any individual cell is causing a problem. 

[anusha_2023]

Thanks for the reply. Yes, I have checked in the same way, and see the below screenshot. The server is not getting started. If I run manually by going into the notebook it works but if I try to schedule server is not picked up. Please check the below screenshot