Forum Discussion

todd-wilson's avatar
todd-wilson
Frequent Visitor
1 year ago
Solved

Cannot reference a Notebook "that attaching" to a different default lakehouse.

I'm running a pipeline starting with a DAG. And getting an error I can't reference a notebook "that attaching to a different default lakehouse". I've tried passing "useRootDefaultLakehouse": True on ...
data pipeline
notebook
  • todd-wilson's avatar
    todd-wilson
    1 year ago

    So I fixed this by adding "useRootDefaultLakehouse": True arg to my dag under my silver script.

     

                "args": {
                    "useRootDefaultLakehouse": True
                }
    Larger snippet...
     {
                "name": "notebook_silver",
                "path": "Notebook Silver",
                "timeoutPerCellInSeconds": 3600,
                "dependencies": ["Notebook Bronze"],
                "args": {
                    "useRootDefaultLakehouse": True
                },
            }
     
    My notebook uses absolute abfss paths to access the bronze datalake and the silver datalake so I think it should have worked according to the docs.
     
    "To specify the location to read from, you can use the relative path if the data is from the default lakehouse of your current notebook. Or, if the data is from a different lakehouse, you can use the absolute Azure Blob File"
     
     
     
Anonymous's avatar
Anonymous
Not applicable
1 year ago

Hi todd-wilson 

 

If you run a notebook directly with below code (not calling by data pipeline), will it run successfully?

%%configure -f
{
    "defaultLakehouse": {
        "name": "<lakehouse name>",
        "id": "<lakehouse id>",
        "workspaceId": "<workspace id>"
    }
}

 

In addition, if you don't use a DAG in the data pipeline, just run multiple notebooks concurrently like below, will it run successfully?

mssparkutils.notebook.runMultiple(["notebook1", "notebook2", "notebook3"])

 

How many notebooks are running concurrently? Is it possible to reduce some notebooks or run them separately to check if this error occurs on a specific notebook or may occur on each notebook? 

 

Best Regards,
Jing

Community Support Team