Forum Discussion
amaaiia
Skilled Sharer
2 years agoUse Dataflow Gen2 for multiple tables ingest dinamically
I have a Data Pipeline to ingest in my Lakehouse several tables located in an on-premise SQL Server, and the tables are going to be daily reloaded incrementally. They have a date field, and every day...
Element115
Memorable Member
2 years agoamaaiia If I understand you correctly, how about having another table in the LH that contains 1 value only, namely the months_to_reload value?
So for each delete-update cycle, in the DFg2, you read in this value in its own M query. Your other queries then can access the value. When the main query is done with its processing, you have another query that updates the months_to_reload value and publishes it back to the LH in replace mode.
Would that do the trick or am I missing something?