Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago

Ingesting on premise data with notebooks

Hi, I have an on premise database with a running PowerBI gateway and this is how we connected to these sources previously. Now, the requierement is that we would recreate our ETL process in Microsoft...
Need Help
pipelines
Anonymous's avatar
Anonymous
Not applicable
2 years ago

Hi Anonymous 
You should use a Data Pipeline to copy the data to lakehouse. Fabric data pipelines will be added to the On Prem data gateway in the next few months. 
What's new and planned for Data Factory in Microsoft Fabric - Microsoft Fabric | Microsoft Learn


If you need something sooner, you should use Azure Data Factory Pipelines with the Self-Hosted Integration Runtime. For more information please refer to these links:
Copy on-premises data using the Azure Copy Data tool - Azure Data Factory | Microsoft Learn
How to ingest data into Fabric using the Azure Data Factory Copy activity - Microsoft Fabric | Microsoft Learn

Hope this helps. Please let me know if you have any further queries.

tmjones2's avatar
tmjones2
Icon for Helper I rankHelper I
2 years ago

So we have to manually create pipeline activities for each table we want? There's really no way to do it in code?

  • smeetsh's avatar
    smeetsh
    Icon for Continued Contributor rankContinued Contributor
    2 years ago

    Create  views on your on prem server, joining the tables you need, if there is a relation between then of course.

    Next use a copy data action in a pipeline to call that view and ingest it into a lakehouse. Thats how we do it for multiple tables at my work.

    I'm not realy sure where the advantage would be to do it with a notebook, since you would (depending on your database structure) have to either call multiple tables in a notebook, use multiple notebooks, or create a join between tables in a notebook.

    Cheers
    Hans