Service Principal Getting "HTTP request forbidden" Error When Reading from Data Warehouse via API-Triggered Notebook
Hi,
I'm experiencing a permissions issue when using a service principal to execute a Fabric notebook via API. The notebook successfully reads from a Lakehouse but fails when trying to read from a Data Warehouse in the same workspace, throwing an "HTTP request forbidden" error.
Environment Details
- Fabric Workspace: Same workspace for notebook, lakehouse, and data warehouse
- Execution Method: API-triggered notebook execution using service principal
- User Roles: I'm admin/creator, colleagues have Contributor role
- Service Principal Role: Contributor role in workspace
Working Scenarios:
- Manual execution by me (admin/creator) - works perfectly
- Manual execution by colleagues with Contributor role - works perfectly
- Service principal querying DW locally (outside of Fabric notebook) - works perfectly
- Service principal reading from Lakehouse in the notebook - works perfectly
Failing Scenario:
- Service principal executing notebook via API and trying to read from Data Warehouse - fails with error
Error Details
ERROR FabricSparkTDSImplicits$FabricSparkTDSRead [Thread-68]: Error processing read request - HTTP request forbidden.
Code snippet
import com.microsoft.spark.fabric
from com.microsoft.spark.fabric.Constants import Constants
view_name = "MyWarehouse.dbo.my_view"
table_name = "MyWarehouse.dbo.my_table"
database_name = "MyWarehouse"
# This line fails when executed by service principal via API
df_table = spark.read.option(Constants.DatabaseName, database_name).synapsesql(f"SELECT * FROM {table_name}")
# or
df_table = spark.read.synapsesql(table_name)
print(f"Table count is {df_table.count()}")
df_view = spark.read.option(Constants.DatabaseName, database_name).synapsesql(f"SELECT * FROM {view_name}")
# or
df_view = spark.read.synapsesql(view_name)
print(f"View count is {df_view.count()}")
About The Service Principal:
In the Azure Service Principal
- Capacity.Read.All
- Lakehouse.ReadWrite.All
- Tenant.Read.All
- Warehouse.ReadWrite.All
- Workspace.Read.All
In the Fabric Warehouse
- SQL Permissions: SELECT, INSERT, UPDATE granted
- Database Roles: db_datareader, db_datawriter
- Additional Access: "Manage Connections and Gateways" granted for the Data Warehouse
Questions
- Is there a specific permission or configuration missing for service principals to access Data Warehouse through Spark in API-triggered notebooks?
- Are there differences in authentication context between manual execution and API-triggered execution that could cause this issue?
- Could this be related to connection pooling or session management when using service principals in Fabric notebooks?
What I've Tried
- Verified all standard permissions are in place
- Confirmed service principal works locally for DW queries
- Confirmed lakehouse access works in the same notebook
- Verified workspace contributor role is assigned
Any insights or suggestions would be greatly appreciated!
Vincent
