Forum Discussion
Streaming dataset
Hello,
I have tried the streaming dataset using API to push data to the dataset from a Databricks notebook, and this works well. However, a message appears: this feature will be retired soon.
Question: how do I use real-time intelligence in Fabric to recreate the same functionality as my streaming dataset?
19 Replies
Hello ollieftw,
Thank you for reaching out to the Microsoft Fabric Forum Community.
The Power BI streaming datasets are being deprecated, you can replicate and enhance this functionality using Real-Time Intelligence in Microsoft Fabric, specifically with Eventstreams and Real-Time Dashboards. Here’s how you can transition:- In the Real-Time Hub, create an Eventstream and use its Apache Kafka endpoint as the target for your Databricks notebook. Update your notebook to push data to this endpoint using a Kafka producer library.
- Route the Eventstream data to an Eventhouse for scalable storage and querying or a Lakehouse for Delta table storage.
- Create a Real-Time Dashboard in Fabric, connect it to the Eventhouse, and use KQL queries to visualize live data, like your Power BI streaming dataset dashboard and Use Fabric Activator to set up real-time alerts for specific data patterns.
For a detailed guide, check out the Real-Time Intelligence tutorial. If you’re using a Custom App to push data, ensure your Fabric capacity is at least F4 to support Eventstreams.
If this information is helpful, please “Accept as solution” and give a "kudos" to assist other community members in resolving similar issues more efficiently.
Thank you.Hello!
I am trying to set up the Apache Kafka you mentioned. Here is a screenshot, what do I put as the values?
Hello ollieftw,
Thanks for sharing the screenshot and your progress on setting up the Apache Kafka connection.
Let's fill in the values for the configuration settings based on your setup with the Eventstream.- Since no existing connection is found, select "New connection" to create a new one. You'll need to provide the Kafka endpoint details from your Databricks environment or Kafka cluster. This is typically provided by your Kafka administrator or Fabric Eventstream settings.
- Enter the specific topic where your Databricks notebook is pushing data. This should match the topic you configured in your notebook's Kafka producer.
- Use a unique identifier for the consumer group, such as eg-data-processing-group, to ensure the Eventstream can process the data stream effectively.
- Leave it as "LATEST" to start consuming from the latest messages, which is suitable if you're setting this up fresh. You can adjust to "EARLIEST" if you need historical data.
- Set it to "SASL_PLAINTEXT" initially. If your Kafka cluster requires encryption, you may need to switch to "SASL_SSL" and provide additional security credentials (e.g: username, password, or certificate) as per your Kafka setup.
For the exact Kafka endpoint and security details, check with your Databricks or Kafka administrator, as these are specific to your environment. Once filled, click "Next" to review and connect. If you encounter any errors, ensure the topic exists and the connection details align with your Databricks notebook configuration.
I trust this addresses your needs. If it does, please “Accept as solution” and give it a "kudos" to help others find it easily.
Thank you.
Hello ollieftw,
Just checking in have you been able to resolve this issue? If so, it would be greatly appreciated if you could mark the most helpful reply accordingly. This helps other community members quickly find relevant solutions.
Please don’t forget to “Accept as Solution” and Give “Kudos” if the response was helpful.
Thank you.Hello!
No I didnt resolve it yet (because our Databricks environment does not have internet access) so I cannot mark anything as “Accept as Solution” and I cannot Give “Kudos”. I will get back to you once I have resolved it.
Thanks
Hello ollieftw,
Could you please confirm if your query has been resolved by the provided solution? If so, please mark it as the solution. This will help other community members solve similar problems faster.
Thank you.
Hello ollieftw,
Could you please confirm if this issue has been resolved? if you need more assistance, please consider reaching out to Microsoft Support. You can provide them with all the steps you've already taken, which will help them understand the issue better and provide a resolution.
Below is the link to create Microsoft Support ticket:
How to create a Fabric and Power BI Support ticket - Power BI | Microsoft LearnHello ollieftw,
Could you please confirm if the issue has been resolved after raising a support case? If a solution has been found, it would be greatly appreciated if you could share your insights with the community. This would be helpful for other members who may encounter similar issues.
Thank you for your understanding and assistance.
Hello!
No, it has not been resolved since we dont have internet connection in our Databricks environment.
Best regards,
Oliver
