Forum Discussion
Streaming dataset
Hello!
I am trying to set up the Apache Kafka you mentioned. Here is a screenshot, what do I put as the values?
Hello ollieftw,
Thanks for sharing the screenshot and your progress on setting up the Apache Kafka connection.
Let's fill in the values for the configuration settings based on your setup with the Eventstream.
- Since no existing connection is found, select "New connection" to create a new one. You'll need to provide the Kafka endpoint details from your Databricks environment or Kafka cluster. This is typically provided by your Kafka administrator or Fabric Eventstream settings.
- Enter the specific topic where your Databricks notebook is pushing data. This should match the topic you configured in your notebook's Kafka producer.
- Use a unique identifier for the consumer group, such as eg-data-processing-group, to ensure the Eventstream can process the data stream effectively.
- Leave it as "LATEST" to start consuming from the latest messages, which is suitable if you're setting this up fresh. You can adjust to "EARLIEST" if you need historical data.
- Set it to "SASL_PLAINTEXT" initially. If your Kafka cluster requires encryption, you may need to switch to "SASL_SSL" and provide additional security credentials (e.g: username, password, or certificate) as per your Kafka setup.
For the exact Kafka endpoint and security details, check with your Databricks or Kafka administrator, as these are specific to your environment. Once filled, click "Next" to review and connect. If you encounter any errors, ensure the topic exists and the connection details align with your Databricks notebook configuration.
I trust this addresses your needs. If it does, please “Accept as solution” and give it a "kudos" to help others find it easily.
Thank you.
Hello!
I got stuck on the first item: I cannot find any Kafka details in the Eventstream Settings, see screenshot.
I then checked my Databricks environment, but there is no Kafka to be seen anywhere in there.
They have partner connect for a lot of services but I cant see any Kafka details:
Here is a list of the Databricks connect partners:
https://www.databricks.com/company/partners/technology
How do I proceed?
Thanks
Oliver
Hello ollieftw,
Thank you for the update and for sharing the screenshots.
It seems there’s some confusion regarding the Kafka setup, which is understandable since the integration with Microsoft Fabric’s Eventstream and Databricks requires specific configuration. Please consider below points:- The Eventstream settings (first screenshot) don’t display Kafka details because the Kafka endpoint is only generated after you fully configure the Eventstream with a data source. Since you haven’t connected a Kafka source yet, the details aren’t visible.
- The Databricks environment (second screenshot) doesn’t show Kafka natively because Kafka isn’t a built-in service in Databricks Partner Connect by default. Databricks supports Kafka integration, but you need to set it up manually or connect to an existing Kafka cluster (e.g., Confluent Cloud, Azure Event Hubs with Kafka compatibility, or a self-managed Kafka instance).
- Check if your organization has an existing Kafka cluster. The Kafka endpoint and topic details would come from this cluster, not directly from Databricks or Fabric.
I trust this information proves useful. If it does, kindly “Accept as solution” and give it a "kudos" to help others locate it easily.
Thank you.Hello!
We dont have a Kafka cluster, unfortunately.
In the streaming dataset one does not need any Kafka cluster, I can just use the API details to push data to the model. Why has Microsoft made it so difficult for me to replicate this easy and simple functionality in Fabric?
What other methods in Fabric can I use to achieve the above?
Thanks.
Oliver