How to stream messages from EventStream to an existing Azure EventHub

roalexan — Mon, 20 May 2024 20:59:44 GMT

Hello all. I'm trying to stream messages from a Fabric EventStream to an existing Azure EventHub. From what I see here, only these four types are supported as EventStream destinations: custom app, KQL database, lakehouse, and reflex. From what I see, custom apps support EventHubs, in the sense that they provide three "protocols": Eventhub, AMQP and Kafka. This would enable callers to make use of the EventHub associated with the custom app.

What I want to do is different. I want the messages sent to the custom app to be streamed to an existing Azure EventHub. Our friend GPT 4 suggests creating an Azure EventGrid and using that as a pub sub message distribution service between the EventHub in the custom app and the existing Azure EventHub. Not having used EventGrid before, I wanted to reach out and see if anyone has any opinions on this approach (or it there are better options / best practices). BTW, I suppose another approach is to write my own code (e.g. Azure function) to read from the custom app's EventHub and write messages to the Azure EventHub, but I'm pretty sure I'd be way better off using a more "off the shelf" solution. Will report back on my findings one way or another.

Re: How to stream messages from EventStream to an existing Azure EventHub

Anonymous — Tue, 02 Jan 2024 05:52:59 GMT

Hi roalexan
Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this.
We will update you once we hear back from them.
Thanks

Re: How to stream messages from EventStream to an existing Azure EventHub

roalexan — Tue, 02 Jan 2024 17:01:48 GMT

FWIW, I for now am using an Azure Function for this - trigger on messages from the source Custom App managed EventHub, then relay messages via code to the Azure EventHub. This works fine. There are pros/cons for Azure Functions versus EventGrid - from our friend chatgpt:

Azure Event Grid and Azure Functions are both cloud services that enable event-driven computing, but they have different purposes and features. Here are some of the pros and cons of each service:

Azure Event Grid is a serverless event bus that allows you to publish and subscribe to events from various sources, such as Azure services, custom applications, or third-party platforms. It supports advanced routing, filtering, and delivery of events to multiple endpoints, such as Azure Functions, Logic Apps, WebHooks, or Event Hubs. Some of the pros of Azure Event Grid are:
- It provides a scalable and reliable infrastructure for event-driven applications, with guaranteed delivery and 99.99% availability.
- It enables enterprise integration and loose coupling of components, as events can be easily consumed by different services without direct dependencies.
- It supports advanced routing and filtering of events based on various criteria, such as subject, data, or custom logic.
- It is cost-effective, as it only charges for the number of events delivered, not the number of events published or the number of subscriptions. Some of the cons of Azure Event Grid are:
- It has a limited number of events per second (10,000 per topic and 5,000 per subscription) and a limited event size (64 KB).
- It does not provide any built-in processing or transformation of events, as it only delivers them to the endpoints.
- It requires additional configuration and management of topics, subscriptions, and endpoints, as well as security and authentication of the event sources and handlers.
Azure Functions is a serverless compute service that allows you to run code in response to events or triggers, such as HTTP requests, timers, queues, or Event Grid. It supports various languages, such as C#, Java, JavaScript, Python, or PowerShell, and various bindings, such as Blob Storage, Cosmos DB, or SignalR. Some of the pros of Azure Functions are:
- It enables rapid development and deployment of event-driven applications, as you only need to write the code and configure the triggers and bindings.
- It provides flexible and dynamic scaling based on the demand and workload, as well as high availability and resiliency of the functions.
- It supports multiple programming models, such as imperative, declarative, or durable, as well as multiple hosting options, such as consumption, premium, or dedicated plans.
- It is cost-efficient, as it only charges for the execution time and memory usage of the functions, not the idle time or the number of instances. Some of the cons of Azure Functions are:
- It has a limited execution time (10 minutes by default, up to 60 minutes on premium or dedicated plans) and a limited memory size (1.5 GB by default, up to 14 GB on premium or dedicated plans).
- It does not support stateful or long-running functions, as it relies on external storage or services to maintain the state or orchestrate the workflow.
- It requires additional testing and monitoring of the functions, as well as error handling and retry logic for the events or triggers.

In summary, Azure Event Grid and Azure Functions are complementary services that can work together to build event-driven applications. Azure Event Grid is responsible for publishing and delivering events, while Azure Functions is responsible for processing and reacting to events. You can use both services to achieve scalability, reliability, and cost-effectiveness of your applications. For more information, you can check out these resources:

topic Re: How to stream messages from EventStream to an existing Azure EventHub in Eventstream

How to stream messages from EventStream to an existing Azure EventHub

Re: How to stream messages from EventStream to an existing Azure EventHub

Re: How to stream messages from EventStream to an existing Azure EventHub