Forum Discussion

amaaiia's avatar
amaaiia
Icon for Skilled Sharer rankSkilled Sharer
2 years ago

Use Dataflow Gen2 for multiple tables ingest dinamically

I have a Data Pipeline to ingest in my Lakehouse several tables located in an on-premise SQL Server, and the tables are going to be daily reloaded incrementally. They have a date field, and every day...
data engineering
Data Integration
power bi
amaaiia's avatar
amaaiia
Icon for Skilled Sharer rankSkilled Sharer
2 years ago

Sorry, maybe I didn't ask the question correctly. Yes, I know that with DFg2 I can do ingests via gateway, in fact, that's how I'm doing it now and I have everything configured to work.

 

My question was, since you tell me to do the ingest with notebooks because otherwise I can't parameterise this process, as the number of tables is variable, can I do the ingest using gateway by notebooks? I think the answer is no, and in that case I understand that the solution you gave me of doing the whole process from the ingestion with notebooks does not work for me.

 

Am I wrong?

Element115's avatar
Element115
Icon for Memorable Member rankMemorable Member
2 years ago

So it looks like you'll have to do everything using the M language in a DFg2 after all.