RealTime File Processing in Microsoft Fabric

Hi ananthkrishna99,

Thank you for confirming that you can use an eventhouse this is an excellent choice for your real-time file processing needs Kudos to lbendlin for the great suggestion. To meet your requirements for automatically processing files as they arrive, sending data to Azure Event Hubs, handling simultaneous file arrivals, and scaling for files ranging from single records to millions, the best approach is to use Eventstream with Eventhouse in Microsoft Fabric. This solution efficiently processes files in near real-time, routes data to Event Hubs, and uses metadata to manage concurrent files and address the Data Activator limitation on parameter passing.

At a high level, Eventstream monitors your file landing location, ingests data into an Eventhouse for real-time analytics, and routes processed data to Event Hubs. Metadata ensures conflict-free processing of simultaneous files, and Eventhouse scales to handle varying file sizes, making it ideal for your proof of concept (POC).

For your reference, official documentation links:

• Eventstream Overview: Learn how to configure real-time data ingestion and routing.
  https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/overview
• Eventhouse in Fabric: Understand Eventhouse for real-time analytics.
  https://learn.microsoft.com/en-us/fabric/real-time-intelligence/eventhouse
• OneLake Shortcuts: Link storage for file ingestion.
  https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts

If this information is helpful, please “Accept as solution” and give a "kudos" to assist other community members in resolving similar issues more efficiently.
Thank you.