Forum Discussion

smpa01's avatar
smpa01
Community Champion
2 years ago

Spark Connector Issue

I am constatntly getting errors when I am trying the following in a notebook (Prod_dw.dbo.factSales is the 3 part name of a Fabric data warehouse table). This notebook is mounted to a lakehouse and DW is not shortcuted to lakehouse.

 

following the Spark Connector documentation (The doc does not mention any shortcuting of DW to LH as a pre req)

 

 

Do you know how can I debug this?

 

 

 

Thank you in advance.

 

 

 

Data Engineering
Notebook

8 Replies

  • frithjof_v's avatar
    frithjof_v
    Community Champion
    2 years ago

    Is your notebook's default Lakehouse in the same workspace as the Prod_dw?

     

    If not, or if your notebook doesn't have a default Lakehouse (look for the pin symbol), then I guess that could be the reason.

     

    Have you tried to use the following code for reading data across workspaces (I guess it could work also within the same workspace):

     

    https://learn.microsoft.com/en-us/fabric/data-engineering/spark-data-warehouse-connector#read-data-across-workspaces

    • smpa01's avatar
      smpa01
      Community Champion
      2 years ago

      Is your notebook's default Lakehouse in the same workspace as the Prod_dw? - yes

       

      if not, or if your notebook doesn't have a default Lakehouse (look for the pin symbol), - yes ; I already mentioned that that notebook is mounted to a lakehouse (LH resides in the same WS as DW)

       

      Have you tried to use the following code for reading data across workspaces - yes, gives following error

      : value synapsesql is not a member of org.apache.spark.sql.DataFrameReader

      • frithjof_v's avatar
        frithjof_v
        Community Champion
        2 years ago

        I tried myself now, and I am getting the following error:

         

        Request to read failed. Reason - HTTP request forbidden.
com.microsoft.spark.fabric.tds.error.FabricSparkTDSInternalAuthError: HTTP request forbidden.

         

         

        So I am not able to use it myself either.
        But to be honest I don't have experience with Scala, so perhaps I did something wrong in my code, I don't know. I tried to follow this page: Spark connector for Microsoft Fabric Synapse Data Warehouse - Microsoft Fabric | Microsoft Learn

         

        I used this code:

         

        Cell 1 (runs fine)

         

        %%spark
import com.microsoft.spark.fabric.tds.implicits.read.FabricSparkTDSImplicits._
import org.apache.spark.sql.functions._

         

         

        Cell 2 (gives the error when I run this cell)

         

        %%spark
val df = spark.read.synapsesql("TestWarehouse.dbo.customer_dimension")

         

         

        I also tried this

        Cell 3 (gives the same error when I run this cell)

         

        %%spark
import com.microsoft.spark.fabric.Constants
val df = spark.read.option(Constants.WorkspaceId, "550e8400-e29b-41d4-a716-446655440000").synapsesql("testwarehouse.dbo.customer_dimension")

         

         

         

        This is my table in the Fabric Warehouse. It's in the same workspace as the Lakehouse which is the Default lakehouse of my Notebook.

        So this is the table I am trying to read in a Notebook by using the Spark connector for Microsoft Fabric Synapse Data Warehouse, but I am getting the error message as mentioned above.

         

         

        I am the user who created (and therefore also admin) of the workspace, the lakehouse and the warehouse.

  • smpa01's avatar
    smpa01
    Community Champion
    2 years ago

    Can someone please shed some light on this topic as to how can I debug this connector?