Forum Discussion

amaaiia's avatar
amaaiia
Icon for Skilled Sharer rankSkilled Sharer
2 years ago

Use Dataflow Gen2 for multiple tables ingest dinamically

I have a Data Pipeline to ingest in my Lakehouse several tables located in an on-premise SQL Server, and the tables are going to be daily reloaded incrementally. They have a date field, and every day...
data engineering
Data Integration
power bi
amaaiia's avatar
amaaiia
Icon for Skilled Sharer rankSkilled Sharer
2 years ago

I've heard that in 1-2 weeks on-prem SQL Server data ingestion with gateway will be available for Copy activity. I guess this update will be so helpful for me, because source and destination can me parametrised in Copy activity. I think I'm going to wait until this feature comes up.

Element115's avatar
Element115
Icon for Memorable Member rankMemorable Member
2 years ago

amaaiia I just stumbled upon this page:  Lakehouse management API - Microsoft Fabric | Microsoft Learn  and read it quickly, but if I am not mistaken, you can load data to a lakehouse table using the lakehouse REST API.  

 

So, couldn't you at the source DB export the data you want in the LH as multiple CSV files; put these files on a secure file server;  use either Python locally to upload the CSV files to the LH, or use Power Automate to do the same?

 

Or if T-SQL allows, do all this from a stored procedure connecting to the LH REST API.

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Icon for Microsoft Employee rankMicrosoft Employee
    2 years ago

    Hi amaaiia 


    We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. In case if you have any resolution please do share that same with the community as it can be helpful to others.
    If you have any question relating to the current thread, please do let us know and we will try out best to help you.
    In case if you have any other question on a different issue, we request you to open a new thread.


    Thanks.