Forum Discussion
Spark Structured Streaming from Azure Event Hubs to multiple Lakehouses (per client isolation)
Hi everyone,
I’m trying to confirm if this architecture is doable in Microsoft Fabric and what the recommended pattern is.
Use case / requirement:
- We ingest streaming data from Azure Event Hubs for 10+ clients.
- Clients require data isolation, so we created one Lakehouse per client (Bronze layer).
- Each event contains a client_id (unique identifier).
- Using Spark Structured Streaming, I want to write each micro-batch to the correct client Lakehouse, into the respective Delta tables, based on client_id.
What I tried
In Fabric notebooks, I can attach a Lakehouse and write to it by hardcoding the destination path. That works only if all data lands in a single Lakehouse. I’m not finding a clean way to dynamically route writes to multiple Lakehouses within the same streaming job.
Questions
- Is it supported / recommended to write from one Spark Structured Streaming job into multiple Lakehouses (different OneLake paths) dynamically based on a column like client_id?
- If yes, what’s the best pattern? For example:
- foreachBatch routing logic writing per client
- multiple streaming queries (one per client)
- any Fabric-native approach I’m missing
3. Are there limitations in Fabric around dynamic writes to multiple Lakehouse items (checkpointing, attached Lakehouse context, permissions, etc.)?
Note
We are intentionally not using Event Streams due to cost/compute concerns, so we moved to Spark Structured Streaming with micro-batches.
If anyone has implemented something similar, I’d really appreciate guidance or a reference architecture.
Thanks!
Hello Jay-RM
Within Fabric notebooks you can write to any Lakehouse by addressing it directly via its OneLake abfss:// path, or by resolving the location using mssparkutils, rather than relying on whichever Lakehouse happens to be attached to the notebook. Spark Structured Streaming fully supports fan‑out using foreachBatch, which allows each micro‑batch — or even subdivisions of a batch — to be routed to different destinations. This means you can direct data to separate Lakehouses or Delta tables according to a value such as client_id.
Microsoft’s guidance also emphasises that “Streaming to Lakehouse” is a supported pattern for Structured Streaming, and it is generally implemented using Spark Job Definitions, ensuring the process continues to run reliably with a retry policy should the underlying infrastructure restart.
Finally, the Lakehouse that is attached to a notebook does not restrict where your writes may go. You are free to write across workspaces by supplying fully‑qualified abfss paths or by programmatically resolving Lakehouse identifiers, giving you full flexibility in multi‑tenant or multi‑workspace scenarios.
You can
- Keep one readStream from Event Hubs.
- In foreachBatch, split the micro‑batch by client_id and write each slice to the target Lakehouse/Delta table using its abfss://{workspaceId}@onelake.dfs.fabric.microsoft.com/{lakehouseId}/Tables/{table}
import sempy.fabric as fabric from notebookutils import mssparkutils clients = { 101: "AcmeCorporation", 102: "ContosoLtd", 103: "FabrikamIndustries", 104: "GlobalGroup" } def get_client_name(client_id: int) -> str: """ Returns the client_name for the supplied client_id. If the client_id does not exist, returns a friendly message. """ return clients.get(client_id, "Unknown client_id") workspace_id = fabric.get_notebook_workspace_id() lh = mssparkutils.lakehouse.get(name=get_client_name(104), workspaceId=workspace_id) lh_path = lh.get("properties").get("abfsPath")
7 Replies
Hello Jay-RM
Within Fabric notebooks you can write to any Lakehouse by addressing it directly via its OneLake abfss:// path, or by resolving the location using mssparkutils, rather than relying on whichever Lakehouse happens to be attached to the notebook. Spark Structured Streaming fully supports fan‑out using foreachBatch, which allows each micro‑batch — or even subdivisions of a batch — to be routed to different destinations. This means you can direct data to separate Lakehouses or Delta tables according to a value such as client_id.
Microsoft’s guidance also emphasises that “Streaming to Lakehouse” is a supported pattern for Structured Streaming, and it is generally implemented using Spark Job Definitions, ensuring the process continues to run reliably with a retry policy should the underlying infrastructure restart.
Finally, the Lakehouse that is attached to a notebook does not restrict where your writes may go. You are free to write across workspaces by supplying fully‑qualified abfss paths or by programmatically resolving Lakehouse identifiers, giving you full flexibility in multi‑tenant or multi‑workspace scenarios.
You can
- Keep one readStream from Event Hubs.
- In foreachBatch, split the micro‑batch by client_id and write each slice to the target Lakehouse/Delta table using its abfss://{workspaceId}@onelake.dfs.fabric.microsoft.com/{lakehouseId}/Tables/{table}
import sempy.fabric as fabric from notebookutils import mssparkutils clients = { 101: "AcmeCorporation", 102: "ContosoLtd", 103: "FabrikamIndustries", 104: "GlobalGroup" } def get_client_name(client_id: int) -> str: """ Returns the client_name for the supplied client_id. If the client_id does not exist, returns a friendly message. """ return clients.get(client_id, "Unknown client_id") workspace_id = fabric.get_notebook_workspace_id() lh = mssparkutils.lakehouse.get(name=get_client_name(104), workspaceId=workspace_id) lh_path = lh.get("properties").get("abfsPath")Hi Jay-RM ,
Thanks for reaching out to the Microsoft fabric community forum.
I would also take a moment to thank deborshi_nag and tayloramy , for actively participating in the community forum and for the solutions you’ve been sharing in the community forum. Your contributions make a real difference.
I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We’re always here to support you.
Best Regards,
Community Support Team
Hi Jay-RM,
My initial response to this was "this would be so much easier with EventStreams".
What you could do is have the Spark Structured Streaming write to different tables in one lakehouse, and then using OneLake Secutiy grant access to each client only to their related tables, and lastly in the "client" lakehouse, make a shortcut to the tables. I think that should achieve your needs.Hi Jay-RM,
If your sastified with the answers you got, please mark the answer as the solution so that future members can easily find it.
I think deborshi_nag's answer is the best on on this thread personally.Thank you for being so gracious. It speaks volumes about your character!
Hi Jay-RM ,
I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We’re always here to support you
Best Regards,
Community Support Team
