Structured Streaming for multiple queries are not picking the next batch after processing the first

Hi Anonymous ,

I have the following two suggestions:

In your code I observe that the processingTime trigger is set to 60 seconds. If new files are added after the trigger interval, they may not be captured immediately. Please try increasing the time interval.

Make sure that new files added to the source directory have unique names and are not overwritten. If the file name is reused, Spark may not detect the change.

If none of this solves your problem, as an alternative, you can recreate a spark job that adds all files (including the new ones) to the source at once.

Best Regards,
Yang
Community Support Team

If there is any post helps, then please consider Accept it as the solution  to help the other members find it more quickly.
If I misunderstand your needs or you still have problems on it, please feel free to let us know. Thanks a lot!