Forum Discussion
Use Dataflow Gen2 for multiple tables ingest dinamically
Sorry, maybe I didn't ask the question correctly. Yes, I know that with DFg2 I can do ingests via gateway, in fact, that's how I'm doing it now and I have everything configured to work.
My question was, since you tell me to do the ingest with notebooks because otherwise I can't parameterise this process, as the number of tables is variable, can I do the ingest using gateway by notebooks? I think the answer is no, and in that case I understand that the solution you gave me of doing the whole process from the ingestion with notebooks does not work for me.
Am I wrong?
EDIT_0: actually, perhaps this might not work due to the huge amount of data that has to be passed around. I don't know the limits put on the HTTP Request action of Power Automate.
------------------------------------------------------------------------------------------------
But I got another idea! How about creating a Power Automate flow that would ingest the data from the on-prem DB, and then inside the pipeline, you would use a Webhook activity to connect to that Power Automate flow and pass the data to your Notebook and do everything in PySpark? That way there is not need to change your firewall rules, as Power Automate uses the gateway.
Calling Microsoft Flow from your application | Power Automate Blog
Chris Webb's BI Blog: Calling Microsoft Flow From Power Query And Power BI (crossjoin.co.uk)
I asked Bing Copilot and got this:
You can use a Fabric pipeline webhook activity to call an endpoint and wait for it to complete2. This webhook can be configured to receive data from a Power Automate flow3.
The Power Automate flow can be set up to get data from an on-premises database1. However, you need to ensure that the necessary ports are open for remote access and securely manage your connection details1.
Remember, the exact steps may vary based on your specific environment and database. Always ensure you’re following best practices for security and compliance in your organization.