Forum Discussion
RealTime File Processing in Microsoft Fabric
Hi ananthkrishna99,
Thank you for confirming that you can use an eventhouse this is an excellent choice for your real-time file processing needs Kudos to lbendlin for the great suggestion. To meet your requirements for automatically processing files as they arrive, sending data to Azure Event Hubs, handling simultaneous file arrivals, and scaling for files ranging from single records to millions, the best approach is to use Eventstream with Eventhouse in Microsoft Fabric. This solution efficiently processes files in near real-time, routes data to Event Hubs, and uses metadata to manage concurrent files and address the Data Activator limitation on parameter passing.
At a high level, Eventstream monitors your file landing location, ingests data into an Eventhouse for real-time analytics, and routes processed data to Event Hubs. Metadata ensures conflict-free processing of simultaneous files, and Eventhouse scales to handle varying file sizes, making it ideal for your proof of concept (POC).
For your reference, official documentation links:
- Eventstream Overview: Learn how to configure real-time data ingestion and routing.
https://learn.microsoft.com/en-us/fabric/real-time-intelligence/event-streams/overview - Eventhouse in Fabric: Understand Eventhouse for real-time analytics.
https://learn.microsoft.com/en-us/fabric/real-time-intelligence/eventhouse - OneLake Shortcuts: Link storage for file ingestion.
https://learn.microsoft.com/en-us/fabric/onelake/onelake-shortcuts
If this information is helpful, please “Accept as solution” and give a "kudos" to assist other community members in resolving similar issues more efficiently.
Thank you.
Event Stream doesn't support all file formats, I want to parse csv, excel, xml and the files are also of very large size which is around 2GB also sometimes. Event stream is for event based processing and not for actual file processing. Databricks Streaming jobs have file watcher capability which is lacking in Fabric. The Data Activator just send notification to the jobs saying a file has arrived. But it can't send any meta data of the which is required for the downstream jobs. Not sure when will these features are planned to release. Please let us know if you have any idea on these.
Hello ananthkrishna99,
Thank you for the additional context around your use case particularly the need to process large CSV, Excel, and XML files (up to 2GB). You are correct that Eventstream and Data Activator in Microsoft Fabric currently have limitations.
To work around these limitations, here’s a hybrid strategy you can implement using Fabric-native tools:
- Can still act as a lightweight trigger to detect new files. Metadata is limited but useful in certain structured landing scenarios.
- Use these to orchestrate downstream processing by capturing arrival events (from Eventstream or another trigger source) and passing file metadata to notebooks as parameters.
- Spark Structured Streaming in Notebooks it handled natively with Spark and Use preprocessing or Python libraries inside the notebook. For large files, Spark supports partitioning and checkpointing to maintain state and avoid conflicts when multiple files arrive concurrently.
- For better scalability, consider converting Excel and XML files to CSV or Parquet before loading into Spark.
- Use Fabric to stream the processed data to Event Hubs for real-time analytics or downstream apps.
Please refer following official documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-streaming-data and Check the Microsoft Fabric Roadmap: https://roadmap.fabric.microsoft.com/?product=administration%2Cgovernanceandsecurity
for updates.We understand the importance of native file watching and metadata-rich triggers. While these features are not yet available in Fabric, they are on the radar for future enhancements. We encourage you to submit ideas in the Fabric Ideas - Microsoft Fabric Community to help prioritize these features.
Hello ananthkrishna99,
Hope everything’s going great on your end! Just checking in has the issue been resolved, or are you still running into problems? Sharing an update can really help others facing the same thing.
Thank you.