Spark Connector Issue

[smpa01]

I am constatntly getting errors when I am trying the following in a notebook (Prod_dw.dbo.factSales is the 3 part name of a Fabric data warehouse table). This notebook is mounted to a lakehouse and DW is not shortcuted to lakehouse.

following the Spark Connector documentation (The doc does not mention any shortcuting of DW to LH as a pre req)

Do you know how can I debug this?

Thank you in advance.

[smpa01]

Can someone please shed some light on this topic as to how can I debug this connector?

[frithjof_v]

Is your notebook's default Lakehouse in the same workspace as the Prod_dw?

If not, or if your notebook doesn't have a default Lakehouse (look for the pin symbol), then I guess that could be the reason.

Have you tried to use the following code for reading data across workspaces (I guess it could work also within the same workspace):

https://learn.microsoft.com/en-us/fabric/data-engineering/spark-data-warehouse-connector#read-data-a...

[smpa01]

Is your notebook's default Lakehouse in the same workspace as the Prod_dw? - yes

if not, or if your notebook doesn't have a default Lakehouse (look for the pin symbol), - yes ; I already mentioned that that notebook is mounted to a lakehouse (LH resides in the same WS as DW)

Have you tried to use the following code for reading data across workspaces - yes, gives following error

: value synapsesql is not a member of org.apache.spark.sql.DataFrameReader

[frithjof_v]

I tried myself now, and I am getting the following error:

Request to read failed. Reason - HTTP request forbidden.
com.microsoft.spark.fabric.tds.error.FabricSparkTDSInternalAuthError: HTTP request forbidden.

So I am not able to use it myself either.
But to be honest I don't have experience with Scala, so perhaps I did something wrong in my code, I don't know. I tried to follow this page: Spark connector for Microsoft Fabric Synapse Data Warehouse - Microsoft Fabric | Microsoft Learn

I used this code:

Cell 1 (runs fine)

%%spark
import com.microsoft.spark.fabric.tds.implicits.read.FabricSparkTDSImplicits._
import org.apache.spark.sql.functions._

Cell 2 (gives the error when I run this cell)

%%spark
val df = spark.read.synapsesql("TestWarehouse.dbo.customer_dimension")

I also tried this

Cell 3 (gives the same error when I run this cell)

%%spark
import com.microsoft.spark.fabric.Constants
val df = spark.read.option(Constants.WorkspaceId, "550e8400-e29b-41d4-a716-446655440000").synapsesql("testwarehouse.dbo.customer_dimension")

This is my table in the Fabric Warehouse. It's in the same workspace as the Lakehouse which is the Default lakehouse of my Notebook.

So this is the table I am trying to read in a Notebook by using the Spark connector for Microsoft Fabric Synapse Data Warehouse, but I am getting the error message as mentioned above.

I am the user who created (and therefore also admin) of the workspace, the lakehouse and the warehouse.

[Luuk_J]

Did you manage to get this to work? I'm running into the same errors with the same setup.

[smpa01]

I did not make any progress, let me know if you make any. I wonder if MS is yet to ship this.

[Luuk_J]

Well, I tried again and everything seems to work fine now 😀
I guess they fixed it, tried restarting sessions and it still works.

The statements in my notebook:

[Luuk_J]

Too bad, I will keep you updated!