Forum Discussion
Streaming dataset
Hello ollieftw,
Thank you for the update and for sharing the screenshots.
It seems there’s some confusion regarding the Kafka setup, which is understandable since the integration with Microsoft Fabric’s Eventstream and Databricks requires specific configuration. Please consider below points:
- The Eventstream settings (first screenshot) don’t display Kafka details because the Kafka endpoint is only generated after you fully configure the Eventstream with a data source. Since you haven’t connected a Kafka source yet, the details aren’t visible.
- The Databricks environment (second screenshot) doesn’t show Kafka natively because Kafka isn’t a built-in service in Databricks Partner Connect by default. Databricks supports Kafka integration, but you need to set it up manually or connect to an existing Kafka cluster (e.g., Confluent Cloud, Azure Event Hubs with Kafka compatibility, or a self-managed Kafka instance).
- Check if your organization has an existing Kafka cluster. The Kafka endpoint and topic details would come from this cluster, not directly from Databricks or Fabric.
I trust this information proves useful. If it does, kindly “Accept as solution” and give it a "kudos" to help others locate it easily.
Thank you.
Hello!
We dont have a Kafka cluster, unfortunately.
In the streaming dataset one does not need any Kafka cluster, I can just use the API details to push data to the model. Why has Microsoft made it so difficult for me to replicate this easy and simple functionality in Fabric?
What other methods in Fabric can I use to achieve the above?
Thanks.
Oliver
Hello ollieftw,
Thanks for your input.
The best option to replicate your API push functionality without a Kafka cluster is the Custom App with Eventstream in Fabric. Because it provides an API-like endpoint (via a connection string) for your Databricks notebook to push data directly, like your streaming dataset, without needing external infrastructure. You can find official guidance in the Microsoft Fabric documentation here: Add a custom endpoint or custom app source to an eventstream - Microsoft Fabric | Microsoft LearnIf this post helps, then please give us ‘Kudos’ and consider Accept it as a solution to help the other members find it more quickly.
Thank you.