topic Re: Enabling Azure Data Factory to Access Restricted REST API and Transfer Data to Lakehouse: IP Address in Pipelines

Enabling Azure Data Factory to Access Restricted REST API and Transfer Data to Lakehouse: IP Address

Anonymous — Fri, 30 Jun 2023 12:18:34 GMT

How can I configure Azure Data Factory to access a REST API and transfer data to a Lakehouse while considering restrictions on IP addresses or domain names? Specifically, I need to allow Azure Data Factory to access the REST API within Fabric, but I have limitations in terms of IP addresses or domain name whitelisting. What steps should I follow to enable this connectivity and ensure data transfer to the Lakehouse? I would appreciate detailed guidance and any specific configurations or settings that need to be applied within Azure Data Factory and the REST API's security settings.

Re: Enabling Azure Data Factory to Access Restricted REST API and Transfer Data to Lakehouse: IP Address

GraceGu — Tue, 04 Jul 2023 01:44:19 GMT

Abhishek_Narain to comment.

Re: Enabling Azure Data Factory to Access Restricted REST API and Transfer Data to Lakehouse: IP Address

Abhishek_Narain — Tue, 04 Jul 2023 16:59:01 GMT

Currently, you can allow-list the whole Azure Data Factory IP range at your source that hosts the REST API. The IP range is shared across Fabric Data Factory. Do discuss this approach with your security team.
https://learn.microsoft.com/en-us/azure/data-factory/azure-integration-runtime-ip-addresses#azure-integration-runtime-ip-addresses-specific-regions (Use the integration runtime IP address). If you want a reduced IP ranges then allow-list only a specific region that your Fabric tenant is setup in.

Alternatively, we are working on adding On-premises gateway support for pipelines in Fabric which will allow you to read data from private rest api sources from within your network or help you with a static IP (on-premises gateway node).

Re: Enabling Azure Data Factory to Access Restricted REST API and Transfer Data to Lakehouse: IP Address

ajarora — Wed, 05 Jul 2023 20:27:54 GMT

Couple of more questions:

1. Do you specifically mean Azure Data Factory or Fabric Data Factory ? The former doesnt support reading/writing into Fabric Lakehouse currently.

2. What is the Fabric API you plan to call for accessing data ?

Re: Enabling Azure Data Factory to Access Restricted REST API and Transfer Data to Lakehouse: IP Address

Anonymous — Mon, 10 Jul 2023 07:54:49 GMT

1. Fabric Data Factory. Based on the answer provided by Abhishek_Narain, the access to the REST API was successful.

Re: Enabling Azure Data Factory to Access Restricted REST API and Transfer Data to Lakehouse: IP Address

ajarora — Mon, 10 Jul 2023 19:45:18 GMT

Curious which API are you looking to call ?

Re: Enabling Azure Data Factory to Access Restricted REST API and Transfer Data to Lakehouse: IP Address

DavidCousins — Mon, 17 Jul 2023 09:41:03 GMT

Nice, thanks for that.

On the reverse, is there anything planned to allow us to control what Data Factory can access? Not sure I'm a fan of allowing it to access any web API on the public internet, especially as it can use them as a data sink.

I popped an idea in here: Restrict web access for Data Factory

Re: Enabling Azure Data Factory to Access Restricted REST API and Transfer Data to Lakehouse: IP Address

Anonymous — Mon, 17 Jul 2023 10:56:19 GMT

Tripadvisor Content API

Re: Enabling Azure Data Factory to Access Restricted REST API and Transfer Data to Lakehouse: IP Address

Anonymous — Mon, 17 Jul 2023 14:12:56 GMT

How can I integrate a web API for data import into ADF after the recent changes? Could you please advise me on the steps to follow?

Re: Enabling Azure Data Factory to Access Restricted REST API and Transfer Data to Lakehouse: IP Address

Anonymous — Tue, 18 Jul 2023 07:42:47 GMT

DavidCousins ajarora What is the idea of having the option in ADF to create web connections and use them as a source in copy pipeline activities since web APIs are no longer allowed?