Consuming Kafka Messages from Kafka Topic Within a VNET using Fabric Pyspark Notebooks

[harshEdwire]

Hello Community,

I’m looking for guidance on the best approach to consume Kafka messages given the following infrastructure:

Current Setup:

• Kafka server is hosted within a VNet inside an AKS (Azure Kubernetes Service) cluster

• Target destination is Microsoft Fabric Lakehouse

• We need to consume Kafka topics from pySpark notebooks to enable custom transformations and land the data into a partitioned table structure (due to the high volume of data in some topics)

Additional Context:

• We've ruled out using Eventstream, as our scenario requires custom data landing logic that it currently doesn't support.

• We're also not using Event Hubs, as we prefer to leverage our existing Kafka deployment within AKS.

While I’ve reviewed this, but it doesn’t address Kafka instances deployed within an AKS cluster specifically.

What I’m looking for:

• Best practices or recommended approaches for securely and efficiently connecting notebooks to Kafka within AKS

• Insights on handling authentication, networking (e.g., private endpoints, VNet integration), and data ingestion patterns suitable for high-throughput workloads

Any guidance, suggestions, or relevant resources would be greatly appreciated!

[v-prasare]

@harshEdwire, As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[v-prasare]

@harshEdwire, As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[v-prasare]

Hi @harshEdwire ,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

[v-prasare · Accepted solution]

Hi @harshEdwire,

please refer to below solved thread and let me know if this helps?

Solved: Consuming Kafka messages from VNET to Fabric LakeH... - Microsoft Fabric Community

Thanks,

Prashanth Are