Solved: ADO.NET Driver for Fabric Data Engineering (Previe...

vensyd

Hi everyone,

I've been exploring the newly released Microsoft ADO.NET Driver for Fabric Data Engineering (v1.0.0) and hit a wall that I can't explain. Posting here in case anyone else has run into this or can shed some light.

---

**What I'm trying to do**

Connect a .NET 8 console app to a Fabric Lakehouse and run Spark SQL CRUD operations via the LivyConnection, using AzureCli auth on macOS.

---

**The error**

Every connection attempt fails at session creation:

```
Microsoft.Spark.Livy.AdoNet.Common.Exceptions.LivyAdoNetException:
Failed to create Livy session: received null response
---> LivyClientException: HTTP 404: Not Found
{"errorCode":"EntityNotFound","message":"The requested resource could not be found"}
at FabricLivyClientImpl.CreateFabricSessionAsync(...)
at FabricSessionBridge.CreateSessionAsync(...)
at LivyConnection.OpenAsync(...)
```

---

**What I've already ruled out**

This is where it gets interesting. The raw Livy API works perfectly fine.

When I POST directly to:
```
https://api.fabric.microsoft.com/v1/workspaces/{ws_id}/lakehouses/{lh_id}/livyapi/versions/2023-12-0...
```
...I get HTTP 202 and a valid session ID back immediately. So:

- Auth is working (AzureCli token is valid)
- Workspace ID is correct (confirmed via API — lists lakehouses successfully)
- Lakehouse ID is correct (confirmed in the API response)
- The Livy API itself is reachable and functional
- Tested against two different Lakehouses in the same workspace — same 404 on both

The driver's internal `CreateFabricSessionAsync` is clearly hitting a different URL path or API version than `livyapi/versions/2023-12-01/sessions`, and that path is returning 404.

---

**Environment**

- Driver: Microsoft.Spark.Livy.AdoNet 1.0.0
- .NET: 8.0.42
- OS: macOS (Apple Silicon / M-series)
- Auth: AzureCli
- Fabric capacity: Trial
- Workspace region: Australia

---

**My hypothesis**

The driver v1.0.0 may be using a different internal Livy API version string or URL structure than what's currently live on the Fabric tenant. Or there's a capacity SKU requirement (Trial vs paid F-SKU) that gates the specific endpoint the driver relies on.

---

**Questions**

1. Has anyone successfully connected using this driver? If so, what capacity type / SKU were you on?
2. Is there a known minimum SKU requirement for the driver's Livy session endpoint to resolve correctly?
3. Is the driver using a different API version internally?

Happy to share full logs with debug-level logging enabled if that helps narrow it down.
Any pointers from the community would be much appreciated.

Thanks

nilendraFabric

Hi

The 404 EntityNotFound is almost certainly your Trial capacity , the ADO.NET driver v1.0.0 appears to target an internal endpoint that just isn't available on Trial SKUs, even though the raw Livy API works fine on them. The two behave differently under the hood.

To quickly confirm, try capturing what URL the driver is actually calling (mitmproxy is handy for this on Mac) , my bet is it's hitting a different path or API version string than livyapi/versions/2023-12-01.

In the meantime, since your raw Livy calls are working perfectly, just use HttpClient directly with AzureCliCredential , you get the exact same result without the driver getting in the way. Not ideal, but it unblocks you while this gets sorted.

Worth raising with Microsoft too , Trial + Australia region + a v1.0.0 preview driver is a pretty specific combo and this could just be a staged rollout gap rather than anything you're doing wrong.

@vensyd

View solution in original post

vensyd

Thanks for the detailed explanation.

Tried again on F16 and finally got to the bottom of it. Used mitmproxy to capture exactly what URL the driver is hitting internally, and the root cause is a URL casing bug in the driver itself.

The driver v1.0.0 is calling:
POST .../lakehouses/{id}/livyApi/versions/2023-12-01/sessions

The correct Fabric API path is:
POST .../lakehouses/{id}/livyapi/versions/2023-12-01/sessions

One character difference -- the driver constructs the URL with livyApi (capital A) but the Fabric REST API is case-sensitive and only resolves livyapi (all lowercase). That's why it returns 404 every time, regardless of whether your workspace ID, lakehouse ID, auth, or capacity are all correct.

To confirm this isn't a config issue, I built a pre-flight checker that runs four checks before the driver attempts to connect -- workspace lookup, capacity check, lakehouse type verification, and a raw Livy API call. All four passed on F16:

[OK] Workspace found: 'Fabric-RnD'
[OK] Lakehouse found: 'Bronze' (type: Lakehouse)
[OK] Livy API reachable (HTTP 202) -- test session created and cleaned up
[FAIL] Driver v1.0.0 -- HTTP 404 at CreateFabricSessionAsync

Same workspace, same lakehouse, same token, same machine. Raw Livy API works. Driver doesn't. mitmproxy showed exactly why.

This is a one-character bug in FabricLivyClientImpl.CreateFabricSessionAsync inside the driver. Nothing the user can work around via connection string or config.

Environment:
- Driver: Microsoft.Spark.Livy.AdoNet v1.0.0
- .NET: 8.0.42
- OS: macOS (Apple Silicon)
- Capacity: F16
- Region: Australia
- Auth: AzureCli

Fix needed: change livyApi to livyapi in the driver's internal URL construction.

Hope this helps the team get a patch out for v1.0.1.

Also attaching a mitmproxy traffic capture showing both calls side by side -- the working raw Livy endpoint (HTTP 202) and the driver's internal call (HTTP 404). The casing difference in the URL path is visible in the capture.

View solution in original post

vensyd