Forum Discussion

smpa01's avatar
smpa01
Community Champion
2 years ago

Spark Connector Issue

I am constatntly getting errors when I am trying the following in a notebook (Prod_dw.dbo.factSales is the 3 part name of a Fabric data warehouse table). This notebook is mounted to a lakehouse and D...
data engineering
Notebook
frithjof_v's avatar
frithjof_v
Community Champion
2 years ago

Is your notebook's default Lakehouse in the same workspace as the Prod_dw?

 

If not, or if your notebook doesn't have a default Lakehouse (look for the pin symbol), then I guess that could be the reason.

 

Have you tried to use the following code for reading data across workspaces (I guess it could work also within the same workspace):

 

https://learn.microsoft.com/en-us/fabric/data-engineering/spark-data-warehouse-connector#read-data-across-workspaces

  • smpa01's avatar
    smpa01
    Community Champion
    2 years ago

    Is your notebook's default Lakehouse in the same workspace as the Prod_dw? - yes

     

    if not, or if your notebook doesn't have a default Lakehouse (look for the pin symbol), - yes ; I already mentioned that that notebook is mounted to a lakehouse (LH resides in the same WS as DW)

     

    Have you tried to use the following code for reading data across workspaces - yes, gives following error

    : value synapsesql is not a member of org.apache.spark.sql.DataFrameReader

    • frithjof_v's avatar
      frithjof_v
      Community Champion
      2 years ago

      I tried myself now, and I am getting the following error:

       

      Request to read failed. Reason - HTTP request forbidden.
com.microsoft.spark.fabric.tds.error.FabricSparkTDSInternalAuthError: HTTP request forbidden.

       

       

      So I am not able to use it myself either.
      But to be honest I don't have experience with Scala, so perhaps I did something wrong in my code, I don't know. I tried to follow this page: Spark connector for Microsoft Fabric Synapse Data Warehouse - Microsoft Fabric | Microsoft Learn

       

      I used this code:

       

      Cell 1 (runs fine)

       

      %%spark
import com.microsoft.spark.fabric.tds.implicits.read.FabricSparkTDSImplicits._
import org.apache.spark.sql.functions._

       

       

      Cell 2 (gives the error when I run this cell)

       

      %%spark
val df = spark.read.synapsesql("TestWarehouse.dbo.customer_dimension")

       

       

      I also tried this

      Cell 3 (gives the same error when I run this cell)

       

      %%spark
import com.microsoft.spark.fabric.Constants
val df = spark.read.option(Constants.WorkspaceId, "550e8400-e29b-41d4-a716-446655440000").synapsesql("testwarehouse.dbo.customer_dimension")

       

       

       

      This is my table in the Fabric Warehouse. It's in the same workspace as the Lakehouse which is the Default lakehouse of my Notebook.

      So this is the table I am trying to read in a Notebook by using the Spark connector for Microsoft Fabric Synapse Data Warehouse, but I am getting the error message as mentioned above.

       

       

      I am the user who created (and therefore also admin) of the workspace, the lakehouse and the warehouse.

      • Luuk_J's avatar
        Luuk_J
        Frequent Visitor
        1 year ago

        Did you manage to get this to work? I'm running into the same errors with the same setup.