Notebook running from data pipeline wants a schema

[bcdobbs]

Hi,

We have a data pipeline that tiggers a series of notebooks. They make use of saveAsTable when writing delta tables.

Pipeline and notebooks haven't changed and have run fine for months however as of yesterday when the pipeline runs the notebooks throw errors about not having a shema. The odd thing is if I run the notebooks individually they work fine.

Is this a change microsofts side ready for lakehouse schema support and what's the best way to fix.

Full error when notebook runs in pipeline context is:

"Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - AnalysisException, Error value - [SCHEMA_NOT_FOUND] The schema `Lakehouse_Dev.LH_Raw.LH_Raw` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a catalog, verify the current_schema() output, or qualify the name with the correct catalog.
To tolerate the error on drop use DROP SCHEMA IF EXISTS.' : "

(Workspace is Lakehouse_Dev, Lakehouse is LH_Raw) The notebook works fine on it's own.

Ben

[Srisakthi]

Hi @bcdobbs ,

Need to explore further. May be Lakehouse Schema is still in preview.

Regards,

Srisakthi

[bcdobbs · Accepted solution]

Yes schema support is very much still in preview but we're not using Schema supported lakehouses. It feels like the data pipeline is making the notebook think the lakehouses are schema supported.

[Srisakthi]

Hi @bcdobbs ,

Yeah its a strange when pipeline suddenly stops running.

In your error message it has LH_Raw.LH_Raw why it is displaying schema name twice ? Is there any change in your pipeline or its parameters?.

'Error name - AnalysisException, Error value - [SCHEMA_NOT_FOUND] The schema `Lakehouse_Dev.LH_Raw.LH_Raw` cannot be found.

Regards,

Srisakthi

[bcdobbs]

Afraid not. Notebook is doing 

saveAsTable(LH_Raw.table_name).

It runs ok on it's own but it feels like when it runs in context of notebook it thinks the lakehouse is schema enabled and suddenly expects workspace.lakehouse.schema. table_name instead