This guide will show you how to create a OneLake shortcut to a VPC-protected Amazon S3 bucket.
Why use the On-premises-data gateway?
Today, organizations are protecting data by leveraging network security capabilities like virtual networks, firewalls and virtual protected clouds (VPC). To access data securely and to provide a bridge between protected environments and Microsoft Fabric, an on-premises data gateway can be used.
Although the name might suggest that the on-premises data gateway can only be used to access your data that is on-premises, it can actually be used to access any data that is protected by any type of firewall or virtual network, including the virtual protected clouds on AWS. More information about the on premises data gateway is available here.
Setting up a gateway is an easy process. You need to provision an EC2 instance within your virtual private cloud; and configure (or open) appropriate ports to securely communicate with Microsoft Fabric. In this tutorial, we will walk you through the steps to complete end-to-end setup.
If you are already using the on-premises data gateway within Fabric for other items, like Pipelines, dataflows or Power BI, you can use the same instance of on-premises data gateway, as long as it also has access to your S3 bucket inside the VPC as shown in the diagram below.
At a high-level, the setup process consists of the following steps:
- Create a public subnet in your VPC environment and assign security groups to the S3 bucket subnet
- Create an EC2 instance within the public subnet
- Install the on-premises data gateway on the EC2 instance
- Open the right ports to Fabric service
- Create a shortcut using the on-premises data gateway
Prerequisites
- A Virtual Private Cloud (VPC) provisioned in AWS. Follow this guide to create a VPC.
- An S3 bucket that is only accessible from within the AWS VPC. Follow this guide to create and secure an S3 bucket to be only accessible from a VPC.
Step-by-step set up
1. Create a public subnet in your VPC environment and assign NSG’s.
- If you don’t have a public subnet follow this guide to create an internet gateway for a subnet in your VPC.
2. Create an EC2 instance within the public subnet
- Install an EC2 instance running the Windows operating system by following these steps. See the minimum installation requirements for the machine SKU and OS version.
- Launch EC2 instance in the public subnet of your VPC. Be sure to save the private key file in a secure place. You will need this in the next step.
3. Install the on-premises data gateway on the EC2 instance
- Connect to your EC2 instance using a Remote Desktop Protocol (RDP) client, following this guide to learn how to connect over RDP. You will need your private key file from the previous step.
- After signing in to the EC2 instance, follow the steps to install the standard OPDG client.
4. Open the right ports to the Fabric service
- If a firewall blocks outbound connections, configure the firewall to allow outbound connections from the gateway to its associated Azure region. The firewall rules on the gateway need to be updated to allow outbound traffic from the gateway server to the following endpoints.
5. Create a shortcut to S3
- Follow this guide to create an Amazon S3 shortcut. Select the OPDG created in the previous steps to ensure the connection goes through the VPC.
