Forum Discussion

ebjim's avatar
ebjim
Resolver I
1 year ago
Solved

Temporary view in a different cell

I have a temporary view created in a cell, which is called in the next cell. 

 

%%sql

 

CREATE OR REPLACE TEMPORARY VIEW vtemp AS

SELECT NAME, CITY 

FROM table1

 

 

import com.microsoft.spark.fabric
from com.microsoft.spark.fabric.Constants import Constants
from pyspark.sql import functions as F
from pyspark.sql import SparkSession
from pyspark.sql.types import IntegerType, StructType, StructField, StringType, DecimalType, DateType
from pyspark.sql.functions import make_date

 

df= spark.sql("SELECT Name FROM vtemp")
 
When I create the view in the same cell (using PySpark), no problem. If I do it in the first (separate) cell, whether it's using Spark SQL and PySpark, I get the View or Table Not Found error. What am I doing wrong?
 
Thanks in advance.
data engineering
Notebook
  • v-menakakota's avatar
    v-menakakota
    1 year ago

    Hi ebjim ,
    Thanks for reaching out to the Microsoft fabric community forum. 

    Spark notebooks commonly show this behavior when working with temporary views across multiple cells. These views are only available within the Spark session in which they were created. In Microsoft Fabric, if a new session starts (which can happen between cell executions), the temporary views are no longer available which is likely why you're seeing the “View or Table not found” error in the next cell.

    To keep things working:

    • Create and use the temporary view within the same cell, or
    • Consider using a global temporary view if you need to access it across cells or sessions.

    If you'd like more detail on how notebook execution and Spark sessions work in Fabric, this Microsoft Learn article is a great resource:
    Develop, execute, and manage notebooks - Microsoft Fabric | Microsoft Learn

    If this post was helpful, please give us Kudos and consider marking Accept as solution to assist other members in finding it more easily.
    Best Regards, 
    Menaka.
    Community Support Team  

2 Replies

  • v-menakakota's avatar
    v-menakakota
    Community Support
    1 year ago

    Hi ebjim ,
    Thanks for reaching out to the Microsoft fabric community forum. 

    Spark notebooks commonly show this behavior when working with temporary views across multiple cells. These views are only available within the Spark session in which they were created. In Microsoft Fabric, if a new session starts (which can happen between cell executions), the temporary views are no longer available which is likely why you're seeing the “View or Table not found” error in the next cell.

    To keep things working:

    • Create and use the temporary view within the same cell, or
    • Consider using a global temporary view if you need to access it across cells or sessions.

    If you'd like more detail on how notebook execution and Spark sessions work in Fabric, this Microsoft Learn article is a great resource:
    Develop, execute, and manage notebooks - Microsoft Fabric | Microsoft Learn

    If this post was helpful, please give us Kudos and consider marking Accept as solution to assist other members in finding it more easily.
    Best Regards, 
    Menaka.
    Community Support Team  

  • ebjim's avatar
    ebjim
    Resolver I
    1 year ago

    Hi v-menakakota ,

     

    Thank you for the helpful response. I do want to ask: how do you create temporary global views in Fabric? Lakehouses and warehouses don't support that.