Dataflow Gen 2 - how to join tables and load results, without loading initial tables?

Hi,

I understand you are trying to load the tables from on-prem to a lakehouse/data warehouse on Fabric.

The option to disable the load we had before was very useful because the dataflow was a starting point to the modeling. Once created, the datasets created over the dataflow would "see" the loaded tables and many times we wouldn't like that. But this doesn't mean that during the refresh processing the tables were not loaded into memory to produce the new one you "modelled".

The Dataflow Gen 2, on the other hand, is used to load the data into the lake, it's not a modelling starting point. In this way, you don't need to bother about "loading" the table. For each table you would like to load into the data lake, you need to set a destination.

So, after merging the tables the way you would like, just set the destination on the resulting table, ignoring the others, and only the resulting table will be loaded into the lake, where in fact the modelling will happen. The "work tables" created in the middle of the ETL will not be on the lake.

But nothing changes in relation to how heavy the processing is. If your concern is about the processing, maybe you would like to consider incremental load options.

Kind Regards,

Dennes