Consuming Kafka Messages from Kafka Topic Within a VNET using Fabric Pyspark Notebooks

Hello Community,

I’m looking for guidance on the best approach to consume Kafka messages given the following infrastructure:

Current Setup:

• Kafka server is hosted within a VNet inside an AKS (Azure Kubernetes Service) cluster

• Target destination is Microsoft Fabric Lakehouse

• We need to consume Kafka topics from pySpark notebooks to enable custom transformations and land the data into a partitioned table structure (due to the high volume of data in some topics)

Additional Context:

• We've ruled out using Eventstream, as our scenario requires custom data landing logic that it currently doesn't support.

• We're also not using Event Hubs, as we prefer to leverage our existing Kafka deployment within AKS.

While I’ve reviewed this, but it doesn’t address Kafka instances deployed within an AKS cluster specifically.

What I’m looking for:

• Best practices or recommended approaches for securely and efficiently connecting notebooks to Kafka within AKS

• Insights on handling authentication, networking (e.g., private endpoints, VNet integration), and data ingestion patterns suitable for high-throughput workloads

Any guidance, suggestions, or relevant resources would be greatly appreciated!