topic Re: Notebook running from data pipeline wants a schema in Pipelines

Notebook running from data pipeline wants a schema

bcdobbs — Sun, 17 Nov 2024 09:59:25 GMT

Hi,

We have a data pipeline that tiggers a series of notebooks. They make use of saveAsTable when writing delta tables.

Pipeline and notebooks haven't changed and have run fine for months however as of yesterday when the pipeline runs the notebooks throw errors about not having a shema. The odd thing is if I run the notebooks individually they work fine.

Is this a change microsofts side ready for lakehouse schema support and what's the best way to fix.

Full error when notebook runs in pipeline context is:

"Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - AnalysisException, Error value - [SCHEMA_NOT_FOUND] The schema `Lakehouse_Dev.LH_Raw.LH_Raw` cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a catalog, verify the current_schema() output, or qualify the name with the correct catalog.
To tolerate the error on drop use DROP SCHEMA IF EXISTS.' : "

(Workspace is Lakehouse_Dev, Lakehouse is LH_Raw) The notebook works fine on it's own.

Ben

Re: Notebook running from data pipeline wants a schema

Srisakthi — Mon, 18 Nov 2024 04:49:59 GMT

Hi bcdobbs ,

Yeah its a strange when pipeline suddenly stops running.

In your error message it has LH_Raw.LH_Raw why it is displaying schema name twice ? Is there any change in your pipeline or its parameters?.

'Error name - AnalysisException, Error value - [SCHEMA_NOT_FOUND] The schema `Lakehouse_Dev.LH_Raw.LH_Raw` cannot be found.

Regards,

Srisakthi

Re: Notebook running from data pipeline wants a schema

bcdobbs — Mon, 18 Nov 2024 06:59:50 GMT

Afraid not. Notebook is doing

saveAsTable(LH_Raw.table_name).

It runs ok on it's own but it feels like when it runs in context of notebook it thinks the lakehouse is schema enabled and suddenly expects workspace.lakehouse.schema. table_name instead

Re: Notebook running from data pipeline wants a schema

Srisakthi — Mon, 18 Nov 2024 07:19:02 GMT

Hi bcdobbs ,

Need to explore further. May be Lakehouse Schema is still in preview.

Regards,

Srisakthi

Re: Notebook running from data pipeline wants a schema

bcdobbs — Mon, 18 Nov 2024 09:06:16 GMT

Yes schema support is very much still in preview but we're not using Schema supported lakehouses. It feels like the data pipeline is making the notebook think the lakehouses are schema supported.