topic How to Call a Notebook mounted with TSQL Analytics from a Notebook running on Spark session? in Data Engineering

How to Call a Notebook mounted with TSQL Analytics from a Notebook running on Spark session?

Shreya_Barhate — Fri, 25 Oct 2024 09:05:41 GMT

I’m currently working on a project where I need to call a notebook mounted with TSQL Analytics from a notebook running on Spark session.

I tried using %run magic command to call the TSQL notebook but I'm facing an issue.

Re: How to Call a Notebook mounted with TSQL Analytics from a Notebook running on Spark session?

Anonymous — Mon, 28 Oct 2024 09:08:45 GMT

Hi Shreya_Barhate

It seems the feature is not well supported yet. When I run a SQL query similar to yours which contains brackets "[]", I got the same error message saying "Syntax error at or near '['".

When I remove all brackets from that query, I got the error "Cannot resolve function 'TOP'..."

However when I run a simple query "SELECT * FROM data2;" it succeeds.

But when the query is "SELECT * FROM mydw.dbo.data2;" it failed with an error that "spark_catalog requires a single-part namespace..."

I guess that as the T-SQL notebook feature is still in preview, it is not very well supported yet and needs to be impoved to work with Spark notebooks better. Please raise your ideas at Fabric Ideas Forum to let the product team get aware of what features you require.

Best Regards,
Jing
If this post helps, please Accept it as Solution to help other members find it. Appreciate your Kudos!

Re: How to Call a Notebook mounted with TSQL Analytics from a Notebook running on Spark session?

JayKidd81 — Fri, 29 Nov 2024 21:49:48 GMT

Related Idea to up-vote! 🙂

Re: How to Call a Notebook mounted with TSQL Analytics from a Notebook running on Spark session?

JayKidd81 — Fri, 29 Nov 2024 21:53:43 GMT

Are there any potential workarounds to run a TSQL Notebook from a PySpark Notebook available? 🙂

We currently have a complex workflow spread across several Notebooks from Lakehouse to Warehouse which we're trying to combine into a single Run-All Notebook. Getting stuck figuring out how to run the TSQL Warehouse Notebooks? 🙂