Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

Notebook data load from a Lakehouse that's NOT default?

I've loaded sources and medallion stages into their own lakehouses in their own workspaces for scalability and trying to keep things tidy from the get-go. I'm now in a position where it would be REALLY useful to, within a notebook, be able to pull data in from a source, run some SQL on it, and load it into a layer. Pretty standard simple transformation step.


I can add those lakehouses to my notebook so they're both referenced there, but I can't get data from any lakehouse that isnt the default for that notebook. If we can't get data from it, why can we add it? Is this somehting that's coming soon or am I doing something wrong?


When I select 'load data' from a table in a lakehouse that isnt default to the notebook, it auto populates the cell, starting with the comment:

 

 

# With Spark SQL, Please run the query onto the lakehouse which is from the same workspace as the current default lakehouse.

df = spark.sql("SELECT * FROM SilverLayerLakehouse.myTable LIMIT 10")
display(df)

 



and trying to run that cell throws the error:

 

AnalysisException: Table or view not found: SilverLayerLakehouse.myTable; line 1 pos 14; 'GlobalLimit 10 +- 'LocalLimit 10 +- 'Project [*] +- 'UnresolvedRelation [SilverLayerLakehouse, myTable], [], false

 


If I switch it to my default lake it runs fine, but I need to load from source and store in silver with some SQL in between. With 'One lake' I assumed my data was visible from anywhere and that was the whole idea behind it, but I can't seem to get my notebook to read anything from a lake in a different workspace? Am I missing something, or is this going to be functionality further down the line?

  • Hi Anonymous You can either write to the Lakehouse using full path or you can create shortcuts in the lakehouse to point to the other lakehouse

     

    Full Path:

     

     

3 Replies

  • AndyDDC's avatar
    AndyDDC
    Icon for Most Valuable Professional rankMost Valuable Professional

    Hi Anonymous You can either write to the Lakehouse using full path or you can create shortcuts in the lakehouse to point to the other lakehouse

     

    Full Path:

     

     

    • Anonymous's avatar
      Anonymous
      Not applicable

      Hi Anonymous 
      We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
      Thanks

    • Anonymous's avatar
      Anonymous
      Not applicable

      Perfect AndyDDC - read locally, write externally. I added the shortcut to my source lake and with a notebook was able to push to my silver layer as you outlined. Thank you