Expert opinion on Incremental refresh choice

Hi fellow community!

I need some expert level advice when choosing what my datasource should be depending on how it behaves with incremental refresh.

Let me explain in detail the situation:

My primary datasource is an Azure blob storage account. Inside this I have a blob container that hold hundreds of text files (each containing hundreds of thousands of lines). The info in this text files is what I combine and transform with powerquery in a typical use case for PowerBI so I can create my reporting Dashboard.

Given the amoun of files present and the amount of info each file contains I originally thought of an architecture where Azure blob storage would be the datasource and I leveraged a dataflow that uses incremental refresh to optimize refreshes and performance. I used This idea  to manage the incremental refresh with a datasource that is not SQL or other "optimized" datasource. 

Originally this worked quite well but for some reason ( I still have the ticket opened with MS support) after including a 4th table (that is not very different from the others in terms of the query complexity) it keeps coming back with refresh errors where apparently it caches to much data ( "PipelineException: The evaluation reached the allowed cache entry size limit" is the error if anyone is curious). Im fairly confident this has to do with the incremental refresh as I removed all the settings for it and the DF refreshed perfectly the whole dataset (which is not even big to begin with since its just a testing environment and right now holds 13GB of data).

Thinking on how I could tackle this problem Im thinking ill split the tasks, one task will be to connect to the datasource and leverage the incremental refresh and once the files are already filtered the dataflow will run a normal full refresh and will do all the usual combining the files, transforming , etc.

I am not sure now if for this initial step I should use a Datamart, a semantic model or another "Staging" Dataflow and was wondering how incremental refresh affects each of them and which one would the most suited for my situation.

Also if anyone has faced this refresh fail issue before and has some insight on how to fix it in the first place (assuming its possible and the fix is not "Upgrade to a higher capacity").

I am running a PPU3 license.

Many thanks in advance for the help!