Can you read a Fabric lakehouse view from spark?

[dwpurdy]

I have a Fabric lake house with a view that I created via the T-SQL endpoint.  Can I query the view from a spark notebook?  If so how?

[martingrosse]

@dwpurdy Hello - I know this has gone cold. But, I think the information you received on this thread is outdated now. While thinking about similar things, I found the following.

There is now a pyspark warehouse connector that lets you access views in the lakehouse.
It could be that you may find this article helps: Spark connector for Microsoft Fabric Data Warehouse - Microsoft Fabric | Microsoft Learn

It could also be that you already know...

[SorenSparso]

Hi,

Has anyone found a solution/workaround for this? Due to the nature of the project, I have to use T-SQL, so writing the view definition SQL directly in the notebook wont work.

Thanks

[puneetvijwani]

@dwpurdy you should be using below way in your notebook cells to create and use that view logic and use as a source in notebook to write out as delta table 

%%sql

CREATE OR REPLACE TEMP VIEW Viewname AS

select statmenents 


PS:If my response helped, kindly select it as the solution. Your kudos are greatly appreciated!

[dwpurdy]

Thanks for the work around, but I was trying to encapusulate transformation logic in T-SQL views and then use the views as sources to write out new delta tables wth a Spark script. 

It's hard to believe it's not possible to read a T-SQL view via the end point in Spark.  Does that also mean, I can't access warehouse tables in Spark via the warehouse SQL end point? 

This seems like a missing feature. In Synapse, it's easy to read a dedicated pool table in Spark.

[govindarajan_d]

It's not possible to read a T-SQL view from Spark. But it's possible to read a warehouse table using Spark. You just need to add the warehouse table as a shortcut and you can access those tables just like the other lakehouse tables. 

[HimanshuS-msft]

Hello @dwpurdy 

I did tried to access the views from a notebook and spark always complained that the object does not exists .

%%sql
SELECT * FROM view1

Error: Table or view not found: view1; line 1 pos 14;
'Project [*]
+- 'UnresolvedRelation [view1], [], false

So i am pretty sure that you cannot access from SPARK  . 

Since views are just SQL query  what you can do as work around is , run the SQL query itself 

%%sql
SELECT * FROM Address

Thanks
HImanshu