Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified in Microsoft Fabric—for free! For a limited time, get a free DP-600 exam voucher to use by the end of 2024. Register now

Reply
somewhereels
Frequent Visitor

Creating views from spark notebook into SQL analytics endpoint?

Hi, 

 

I'm very new to data engineering and I'm having some trouble creating views from a spark notebook into the default lakehouse. The code runs successfully and tells me a view is being created but it doesn't appear in the lakehouse SQL endpoint.

 

Any tips would be appreciated.

1 ACCEPTED SOLUTION
frithjof_v
Community Champion
Community Champion

You can create the views directly in the Lakehouse's SQL Analytics Endpoint.

View solution in original post

8 REPLIES 8
frithjof_v
Community Champion
Community Champion

You can create the views directly in the Lakehouse's SQL Analytics Endpoint.

Thankyou, I was worried this was the case. I was hoping that I could use spark notebooks because I cant automate the run of view scripts in a lakehouse via a pipeline.

Anonymous
Not applicable

Hi @somewhereels ,

If you are looking for a method to query the lakehouse from fabric pipeline then you can achieve a similar result using a script activity.

Take the SQL analytics endpoint from the lakehouse to create a new cloud connection - use External as an option and run the query against the cloud connection within the pipeline. Just ran and works.

Steps to connect to Lakehouse Endpoint using Script Activity:

1. Copy the SQL Endpoint from Lakehouse.

vgchennamsft_0-1712604512600.png


2. In Script, click on External and then Azure SQL Database, fill the details and use Organizational Account to establish connection.

vgchennamsft_1-1712604524834.png


3. Check the connection 

vgchennamsft_2-1712604547304.png


4. Execute it

vgchennamsft_3-1712604557926.png


In your case, if you want to create a view,

vgchennamsft_0-1712637361735.png

Result:

vgchennamsft_1-1712637410310.png

 


FYI: Support for SQL endpoint for Lakehouse is in our backlog but we don't have a ETA.

Hope this is is helpful. Please let me know incase of further queries.





Anonymous
Not applicable

Hi @somewhereels ,

We haven’t heard from you on the last response and was just checking back to see if workaround helped you?

Hi yes I can get this solution to work, thankyou for your time. I don't think I tried this because I was under the assumption that the sql endpoint was read-only? 

 

Anonymous
Not applicable

Hi @somewhereels ,

Glad to know that you got some insights. Please continue using Fabric Community on your further queries.

Anonymous
Not applicable

Hi @somewhereels ,

Thanks for using Fabric Community.

I can also observe the similar behaviour. Unfortunately I cannot access/see the views in SQL Endpoint,  created through Fabric Notebook. At present we can only see the tables in SQL Endpoint, when created through Fabric Notebook.

At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Anonymous
Not applicable

Hi @somewhereels ,

Internal Team confirmed that this is not any issue.

Spark views won't show up in the SQL endpoint. It is the same behavior with Synapse Spark too. Views created in Synapse won't show up in Synapse serverless.



As @frithjof_v  said, if you are interested to create a view in lakehouse, you can use sql endpoint functionality.

Hope this is helpful. Please let me know incase of further queries.


Helpful resources

Announcements
November Carousel

Fabric Community Update - November 2024

Find out what's new and trending in the Fabric Community.

Live Sessions with Fabric DB

Be one of the first to start using Fabric Databases

Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.

November Update

Fabric Monthly Update - November 2024

Check out the November 2024 Fabric update to learn about new features.

Las Vegas 2025

Join us at the Microsoft Fabric Community Conference

March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Early Bird pricing ends December 9th.