topic Re: Spark Connector Issue in Data Engineering

Spark Connector Issue

smpa01 — Mon, 22 Jul 2024 03:47:23 GMT

I am constatntly getting errors when I am trying the following in a notebook (Prod_dw.dbo.factSales is the 3 part name of a Fabric data warehouse table). This notebook is mounted to a lakehouse and DW is not shortcuted to lakehouse.

following the Spark Connector documentation (The doc does not mention any shortcuting of DW to LH as a pre req)

Do you know how can I debug this?

Thank you in advance.

Re: Spark Connector Issue

frithjof_v — Mon, 22 Jul 2024 05:45:07 GMT

Is your notebook's default Lakehouse in the same workspace as the Prod_dw?

If not, or if your notebook doesn't have a default Lakehouse (look for the pin symbol), then I guess that could be the reason.

Have you tried to use the following code for reading data across workspaces (I guess it could work also within the same workspace):

https://learn.microsoft.com/en-us/fabric/data-engineering/spark-data-warehouse-connector#read-data-across-workspaces

Re: Spark Connector Issue

smpa01 — Mon, 22 Jul 2024 18:44:44 GMT

Is your notebook's default Lakehouse in the same workspace as the Prod_dw? - yes

if not, or if your notebook doesn't have a default Lakehouse (look for the pin symbol), - yes ; I already mentioned that that notebook is mounted to a lakehouse (LH resides in the same WS as DW)

Have you tried to use the following code for reading data across workspaces - yes, gives following error

: value synapsesql is not a member of org.apache.spark.sql.DataFrameReader

Re: Spark Connector Issue

frithjof_v — Mon, 22 Jul 2024 14:30:47 GMT

I tried myself now, and I am getting the following error:

Request to read failed. Reason - HTTP request forbidden. com.microsoft.spark.fabric.tds.error.FabricSparkTDSInternalAuthError: HTTP request forbidden.

So I am not able to use it myself either.
But to be honest I don't have experience with Scala, so perhaps I did something wrong in my code, I don't know. I tried to follow this page: Spark connector for Microsoft Fabric Synapse Data Warehouse - Microsoft Fabric | Microsoft Learn

I used this code:

Cell 1 (runs fine)

%%spark import com.microsoft.spark.fabric.tds.implicits.read.FabricSparkTDSImplicits._ import org.apache.spark.sql.functions._

Cell 2 (gives the error when I run this cell)

%%spark val df = spark.read.synapsesql("TestWarehouse.dbo.customer_dimension")

I also tried this

Cell 3 (gives the same error when I run this cell)

%%spark import com.microsoft.spark.fabric.Constants val df = spark.read.option(Constants.WorkspaceId, "550e8400-e29b-41d4-a716-446655440000").synapsesql("testwarehouse.dbo.customer_dimension")

This is my table in the Fabric Warehouse. It's in the same workspace as the Lakehouse which is the Default lakehouse of my Notebook.

So this is the table I am trying to read in a Notebook by using the Spark connector for Microsoft Fabric Synapse Data Warehouse, but I am getting the error message as mentioned above.

I am the user who created (and therefore also admin) of the workspace, the lakehouse and the warehouse.

Re: Spark Connector Issue

smpa01 — Mon, 29 Jul 2024 01:58:55 GMT

Can someone please shed some light on this topic as to how can I debug this connector?

Re: Spark Connector Issue

Luuk_J — Mon, 19 Aug 2024 10:44:12 GMT

Did you manage to get this to work? I'm running into the same errors with the same setup.

Re: Spark Connector Issue

smpa01 — Mon, 19 Aug 2024 13:17:32 GMT

I did not make any progress, let me know if you make any. I wonder if MS is yet to ship this.

Re: Spark Connector Issue

Luuk_J — Tue, 20 Aug 2024 12:13:50 GMT

Too bad, I will keep you updated!

Re: Spark Connector Issue

Luuk_J — Fri, 23 Aug 2024 08:29:55 GMT

Well, I tried again and everything seems to work fine now 😀
I guess they fixed it, tried restarting sessions and it still works.

The statements in my notebook: