RealTime File Processing in Microsoft Fabric

Hello ananthkrishna99,

Thank you for the additional context around your use case particularly the need to process large CSV, Excel, and XML files (up to 2GB). You are correct that Eventstream and Data Activator in Microsoft Fabric currently have limitations.

To work around these limitations, here’s a hybrid strategy you can implement using Fabric-native tools:

• Can still act as a lightweight trigger to detect new files. Metadata is limited but useful in certain structured landing scenarios.
• Use these to orchestrate downstream processing by capturing arrival events (from Eventstream or another trigger source) and passing file metadata to notebooks as parameters.
• Spark Structured Streaming in Notebooks it handled natively with Spark and Use preprocessing or Python libraries inside the notebook. For large files, Spark supports partitioning and checkpointing to maintain state and avoid conflicts when multiple files arrive concurrently.
• For better scalability, consider converting Excel and XML files to CSV or Parquet before loading into Spark.
• Use Fabric to stream the processed data to Event Hubs for real-time analytics or downstream apps.

Please refer following official documentation: https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-streaming-data and Check the Microsoft Fabric Roadmap: https://roadmap.fabric.microsoft.com/?product=administration%2Cgovernanceandsecurity
for updates.

We understand the importance of native file watching and metadata-rich triggers. While these features are not yet available in Fabric, they are on the radar for future enhancements. We encourage you to submit ideas in the Fabric Ideas - Microsoft Fabric Community to help prioritize these features.