Forum Discussion

somewhereels's avatar
somewhereels
Frequent Visitor
2 years ago
Solved

Creating views from spark notebook into SQL analytics endpoint?

Hi, 

 

I'm very new to data engineering and I'm having some trouble creating views from a spark notebook into the default lakehouse. The code runs successfully and tells me a view is being created but it doesn't appear in the lakehouse SQL endpoint.

 

Any tips would be appreciated.

data engineering
  • frithjof_v's avatar
    frithjof_v
    2 years ago

    You can create the views directly in the Lakehouse's SQL Analytics Endpoint.

9 Replies

  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi somewhereels ,

    Thanks for using Fabric Community.

    I can also observe the similar behaviour. Unfortunately I cannot access/see the views in SQL Endpoint,  created through Fabric Notebook. At present we can only see the tables in SQL Endpoint, when created through Fabric Notebook.

    At this time, we are reaching out to the internal team to get some help on this .
    We will update you once we hear back from them.

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi somewhereels ,

      Internal Team confirmed that this is not any issue.

      Spark views won't show up in the SQL endpoint. It is the same behavior with Synapse Spark too. Views created in Synapse won't show up in Synapse serverless.


      As frithjof_v  said, if you are interested to create a view in lakehouse, you can use sql endpoint functionality.

      Hope this is helpful. Please let me know incase of further queries.


  • frithjof_v's avatar
    frithjof_v
    Community Champion
    2 years ago

    You can create the views directly in the Lakehouse's SQL Analytics Endpoint.

    • somewhereels's avatar
      somewhereels
      Frequent Visitor
      2 years ago

      Thankyou, I was worried this was the case. I was hoping that I could use spark notebooks because I cant automate the run of view scripts in a lakehouse via a pipeline.

      • Anonymous's avatar
        Anonymous
        Not applicable
        2 years ago

        Hi somewhereels ,

        If you are looking for a method to query the lakehouse from fabric pipeline then you can achieve a similar result using a script activity.

        Take the SQL analytics endpoint from the lakehouse to create a new cloud connection - use External as an option and run the query against the cloud connection within the pipeline. Just ran and works.

        Steps to connect to Lakehouse Endpoint using Script Activity:

        1. Copy the SQL Endpoint from Lakehouse.


        2. In Script, click on External and then Azure SQL Database, fill the details and use Organizational Account to establish connection.


        3. Check the connection 


        4. Execute it


        In your case, if you want to create a view,

        Result:

         


        FYI: Support for SQL endpoint for Lakehouse is in our backlog but we don't have a ETA.

        Hope this is is helpful. Please let me know incase of further queries.