Blog Post

Data Engineering Community Blog
2 MIN READ

A reference architecture for private API access from Microsoft Fabric

dimkalamaras's avatar
dimkalamaras
Icon for Microsoft Employee rankMicrosoft Employee
4 months ago

Fabric notebooks often need to read from authenticated REST APIs that live inside a network boundary — historians, MES, custom microservices, on-prem inventory systems. Public IPs are off the table, Fabric can't host a VPN client, and copying the data into the lakehouse first defeats the point.

This repo is a reference architecture for the path that actually works in production: Fabric → Managed Private Endpoint → Azure Function App (acting as a thin reverse proxy) → N private backends — with no changes to the backend code.

 

 

The shape

The Function App is a ~100-line Python proxy on Flex Consumption, VNet-integrated:

  • One catch-all route /<backend>/<anything> — the first URL segment selects the backend, everything after is forwarded unchanged.
  • Server-side credentials. The proxy adds Basic auth outbound; Fabric never sees or sends backend secrets.
  • Response URL rewriting so HATEOAS-style links stay self-consistent and callable.
  • Adding a backend = one app setting + one line of code.

Why it fits Fabric

  • MPE is Fabric's native way to reach a private Azure resource; once approved, Fabric resolves the Function App's hostname to a private IP on Microsoft's backbone.
  • Flex Consumption gives you VNet integration without an App Service Environment and scales per second.
  • No backend changes — the backends keep speaking plain HTTP/Basic-auth on the VNet; they don't know the caller is Fabric.

What's in the repo

End-to-end walkthrough: VNet + subnets, two FastAPI VMs standing in for on-prem servers, the proxy itself, Flex Consumption deployment with VNet integration, storage hardening (the Microsoft.Storage service endpoint gotcha that costs everyone half a day), MPE creation and approval, the Fabric notebook, and a CI/CD path via Azure Repos + Deployment Center.

 

Repo: DimKal-Org/fabric-mpe-function-proxy - key files: README.md, src/function_app.py, notebooks/fabric_call_function.ipynb.

 

 

Published 4 months ago
Version 1.0

3 Comments

  • Great reference architecture dimkalamaras 

    The Managed Private Endpoint + Function App as a reverse proxy pattern is exactly what we needed for connecting Fabric Notebooks to on-prem APIs without exposing public IPs.

    Loved the "one catch-all route" and server-side auth approach. Keeps Fabric from ever seeing backend secrets.

    Going to try this with our MES system. Quick question: Did you face any cold-start issues with Flex Consumption when hitting multiple backends in parallel?

    Thanks for sharing the repo!

  • Thanks dimkalamaras for the clarification! 

     

    Good to know about the "always on" instances in Flex Consumption. That solves the cold-start concern for our MES system.

     

    Will implement this MPE + Function App proxy pattern and share results. Appreciate you open-sourcing the repo!