Forum Discussion

MartijnCyg's avatar
MartijnCyg
Frequent Visitor
1 year ago
Solved

How to connect to an external API data source that needs ip whitelisting?

I'm trying to connect to an external API that requires IP whitelisting, but its challenging because Fabric uses a range of IP addresses that change dynamically. Its not an option to whitelist the who...
  • iainsc's avatar
    1 year ago

    We too had this issue, and have solved it relatively simply in the end.

    The idea is to set up a VNet Data Gateway in Azure, then put it in a subnet fronted with a NAT Gateway which can be configured to have whatever IP address you need.

     

    1. Create a VNet Data Gateway: https://learn.microsoft.com/en-us/data-integration/vnet/create-data-gateways
    2. Create the NAT Gateway and give it a fixed IP address: https://learn.microsoft.com/en-us/azure/nat-gateway/quickstart-create-nat-gateway-portal
    3. Create a Web/HTTPS activity from Fabric and route it via the VNet Data Gateway: 
  • MartijnCyg's avatar
    MartijnCyg
    1 year ago

    Hi, 

     

    Eventually all proposed solutions did not work for us, but we figured it out with a very simple solution, logic app!

     

    what we did:

    1. Logic app:
      • Create a logic app in Azure with a "When an HTTP request is received" trigger . We added a JSON schema for the dynamic parts from our notebook in Fabric (e.g. endpoints and auth token)
      • Add HTTP action and built with the dynamic parts from the above trigger
      • Add response action.
    2. Then we whitelisted all outgoing ip adresses from the logic app (found under settings > properties) in the external API 
    3. Lastly, we put the logic app url in our fabric notebook and called it with this line of code:
     response = session.post(logic_app_url, headers=headers, data=json.dumps(payload))

     

    Our notebook script loops through the API endpoints, calls the Logic App for each one, and writes the results directly to a Delta table and this works perfectly fine!!

     

    Hopefully this helps other people within the (near) future. I struggled with this for a very long time.