Kafka -> ? -> PowerBI

[LOSt_210]

Hello,

I have a case for which I need solution suggestions. The requirements are:

1. Retrieve data from different Kafka topics, where authorization is required.
2. Store the Kafka data so that I can use historical data in the report.
3. Have a scalable solution that can handle 200,000+ data points from Kafka each day in the future.
4. Easily access the data to build Power BI reports.

I have read a bit about Azure Event Hubs and Azure Stream Analytics. Could this be a viable approach? Any experiences with best practices for handling data from Kafka?

[LOSt_210]

great answer - thank you!

[V-yubandi-msft · Accepted solution]

Hi @LOSt_210 ,

1. Event Hubs supports Kafka, enabling seamless data ingestion from existing Kafka producers.

2. Utilize the Capture feature to store Kafka data in Azure Data Lake Storage Gen2 or Azure Blob Storage. For processing, use Azure Stream Analytics and store results in Azure SQL Database or Azure Cosmos DB.

3. Handles 200,000 events/day. Start with Standard tier namespace + 1 to 2 Throughput Units and enable Auto Inflate for dynamic scaling. Use 1 to 3 Streaming Units and 4 to 8 partitions for optimal parallel processing. Azure Data Lake scales efficiently using date-based partitioning for optimized retrieval.

4. Send processed summaries from Stream Analytics to Power BI Streaming Datasets for live dashboards.

Power BI connects to Azure Data Lake Storage and Azure SQL Database.

Azure Data Factory can format and clean raw data before reporting.

FYI:

If my response solved your query, please mark it as the Accepted solution to help others find it easily.

And if my answer was helpful, I'd really appreciate a 'Kudos'.