Eventstream read EventHub with Avro format and Schema Registry (not schema embedded)

[khoa_nguyenanh]

Hi all,

I am having an issue with the Eventstream as below scenarior. Really appriciate any help.

In the project, the application send messages to EventHub with Avro format and using Schema Registry (Json schema) method 

The Schema Registry have an ID and linked to EventHub message

Then I configed an Eventstream to read data from that Eventhub then got this error.

I also tried with embedded schema in eventhub message and eventstream can read without issue.

I know Microsoft Fabric Eventstream still not support Schema Registry Avro format method (only support with embedded schema now), but I need to know any plan to release this support? or any workaround with realtime loading as same as eventstream?

Thanks in advance.

[khoa_nguyenanh]

Hi all,

Just to update the solution for others to consider later: we agreed to use Azure Function. While it introduces a slight delay compared to EventStream, it is a much more cost-effective solution.

Cheers.

[v-hashadapu]

Hi @khoa_nguyenanh , Thanks for the update and sharing the information. This will help the community a lot.

[khoa_nguyenanh]

Thanks you, Yes, Please close this thread.

There are some work around such as using Azure function or convert the EventHub to embedded schema but it's not acepted due to the real time requirement. So, it's pending at the moment.

Thanks again.

[v-hashadapu]

Hi @khoa_nguyenanh , thanks for the update. If you have any other queries, please feel free to make a new post. We are always happy to help.

[v-hashadapu]

Hi @khoa_nguyenanh ,It has been some time since we heard back from you. According to our guidelines, we need to close this thread. If possible, could you please provide the details of your workaround or an estimated time of arrival? Thank you.

[v-hashadapu]

Hi @khoa_nguyenanh , Just getting back to see if your issue is solved. If it did, please share the insights here and consider marking them "Accept as Solution" as this will help others with similar issues find the solution easily.

Thank you.

[v-hashadapu]

Hi @khoa_nguyenanh , hope you are doing well, Just wanted to confirm, if your issue is solved as you said you are discussing a workaround. If so, kindly share the details and mark it or any other helpful replies "Accept as Solution", this will help others with similar queries find the answer easily. 
Thank you.

[khoa_nguyenanh]

Hi,

This is still under discussion, and a final decision hasn't been reached. We'll provide an update here when it's available.

Thanks

[v-hashadapu]

Hi @khoa_nguyenanh , Thank you. we appreciate it. 

[khoa_nguyenanh]

Thanks for your confirmation.

We actually implemented Azure Function for other feature which is not required realtime data. 

However, in this scenarior, it required realtime solution which can provided by Eventstream. 

The embeded schema solution is an old approach. While the Schema registry is newer solution with more advantages provided by Microsoft.

Anyway, Thanks again. Let me discuss with the team for a workaround solution.

[v-hashadapu]

Hi @khoa_nguyenanh , thanks for the update. We would be glad to hear your workaround. Could you please share the details here and mark it as 'Accept as solution' so others with similar issues may find it easily? If you have any other questions, feel free to post them (as a new post) in the community. We're always happy to assist.

Thank you.

[v-hashadapu · Accepted solution]

Hi @khoa_nguyenanh , Thank you for reaching out to the Microsoft Community Forum.

As you observed, when Avro messages include the schema directly embedded in the payload, the pipeline will be read and process the data without any issues. This embedded schema approach is, as of now, the only supported method for Avro decoding in Eventstream. Microsoft has not yet announced support for resolving external schemas from Azure Schema Registry within Eventstream and there’s nothing on the public roadmap or in the documentation to suggest that this capability is coming soon. So, while it’s a reasonable expectation, it’s not something you can plan around at this stage.

If changing the producing application to embed schemas isn’t possible, there’s still a practical workaround. You can implement a lightweight middleware, typically using an Azure Function or Stream Analytics job that reads from your existing Event Hub, deserializes the Avro message using the Schema Registry, then re-serializes it with the schema embedded in the payload. That transformed message can be forwarded to another Event Hub, which Fabric Eventstream can consume.

If this helped solve the issue, please consider marking it “Accept as Solution” so others with similar queries may find it more easily. If not, please share the details, always happy to help.
Thank you.