Streaming dataset

Hello ollieftw,

Thanks for sharing the screenshot and your progress on setting up the Apache Kafka connection.

Let's fill in the values for the configuration settings based on your setup with the Eventstream.

• Since no existing connection is found, select "New connection" to create a new one. You'll need to provide the Kafka endpoint details from your Databricks environment or Kafka cluster. This is typically provided by your Kafka administrator or Fabric Eventstream settings.
• Enter the specific topic where your Databricks notebook is pushing data. This should match the topic you configured in your notebook's Kafka producer.
• Use a unique identifier for the consumer group, such as eg-data-processing-group, to ensure the Eventstream can process the data stream effectively.
• Leave it as "LATEST" to start consuming from the latest messages, which is suitable if you're setting this up fresh. You can adjust to "EARLIEST" if you need historical data.
• Set it to "SASL_PLAINTEXT" initially. If your Kafka cluster requires encryption, you may need to switch to "SASL_SSL" and provide additional security credentials (e.g: username, password, or certificate) as per your Kafka setup.

For the exact Kafka endpoint and security details, check with your Databricks or Kafka administrator, as these are specific to your environment. Once filled, click "Next" to review and connect. If you encounter any errors, ensure the topic exists and the connection details align with your Databricks notebook configuration.

I trust this addresses your needs. If it does, please “Accept as solution” and give it a "kudos"  to help others find it easily.
Thank you.