RealTime File Processing options in Microsoft Fabric

Hi ananthkrishna99 

Thank you for reaching out to the Microsoft Fabric Community Forum.

The optimal solution integrates Spark Structured Streaming for real-time file detection and processing, Fabric Pipelines for orchestration, and Eventstream for seamless EventHub integration. This approach avoids Data Activator's limitations, manages concurrent files safely, and scales for varying file sizes.

To set up the Lakehouse and landing folder, create a Lakehouse in your Fabric workspace, define a folder structure in the Files section for source files, and create a Delta table in the Tables section to temporarily store processed data before routing to EventHub.

To utilize Spark Structured Streaming for real-time file detection and processing, create a new Notebook in Fabric's Data Engineering workload and use PySpark to establish a streaming job monitoring the Files/landing/ folder.

Create an Eventstream item in your Fabric workspace, add a Lakehouse source to read new records from the Delta table, and connect to an Azure Event Hub instance to map the Delta table columns to the Event Hub message format.

Then, orchestrate with Fabric Pipelines by creating a pipeline in the Data Factory workload, adding activities to trigger the Spark Structured Streaming Notebook and Eventstream processing, and scheduling the pipeline to run continuously or on a set schedule.

Finally, optimize for file size variability by configuring Spark Structured Streaming to handle both small and large files efficiently and monitor the process using Fabric Monitoring Hub and Azure Portal.

If this response resolves your query, please mark it as the Accepted Solution to assist other community members. A Kudos is also appreciated if you found the response helpful.

Thank You!