topic Re: Consuming Kafka Messages from Kafka Topic Within a VNET using Fabric Pyspark Notebooks in Data Engineering

Consuming Kafka Messages from Kafka Topic Within a VNET using Fabric Pyspark Notebooks

harshEdwire — Tue, 13 May 2025 10:05:49 GMT

Hello Community,

I’m looking for guidance on the best approach to consume Kafka messages given the following infrastructure:

Current Setup:

Kafka server is hosted within a VNet inside an AKS (Azure Kubernetes Service) cluster
Target destination is Microsoft Fabric Lakehouse
We need to consume Kafka topics from pySpark notebooks to enable custom transformations and land the data into a partitioned table structure (due to the high volume of data in some topics)

Additional Context:

We've ruled out using Eventstream, as our scenario requires custom data landing logic that it currently doesn't support.
We're also not using Event Hubs, as we prefer to leverage our existing Kafka deployment within AKS.

While I’ve reviewed this, but it doesn’t address Kafka instances deployed within an AKS cluster specifically.

What I’m looking for:

Best practices or recommended approaches for securely and efficiently connecting notebooks to Kafka within AKS
Insights on handling authentication, networking (e.g., private endpoints, VNet integration), and data ingestion patterns suitable for high-throughput workloads

Any guidance, suggestions, or relevant resources would be greatly appreciated!

Re: Consuming Kafka Messages from Kafka Topic Within a VNET using Fabric Pyspark Notebooks

v-prasare — Wed, 14 May 2025 10:32:07 GMT

Hi harshEdwire,

please refer to below solved thread and let me know if this helps?

Solved: Consuming Kafka messages from VNET to Fabric LakeH... - Microsoft Fabric Community

Thanks,

Prashanth Are

Re: Consuming Kafka Messages from Kafka Topic Within a VNET using Fabric Pyspark Notebooks

v-prasare — Tue, 20 May 2025 04:29:03 GMT

Hi harshEdwire ,

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

Re: Consuming Kafka Messages from Kafka Topic Within a VNET using Fabric Pyspark Notebooks

v-prasare — Mon, 26 May 2025 02:40:33 GMT

harshEdwire, As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query

Re: Consuming Kafka Messages from Kafka Topic Within a VNET using Fabric Pyspark Notebooks

v-prasare — Thu, 29 May 2025 04:18:58 GMT

@harshEdwire, As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided for your issue worked? or let us know if you need any further assistance here?

Thanks,

Prashanth Are

MS Fabric community support

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly and give Kudos if helped you resolve your query