topic Re: Pipeline - Copy Data - Amazon Redshift - No gateway option in Pipelines

Pipeline - Copy Data - Amazon Redshift - No gateway option

coverturfLSH — Tue, 15 Oct 2024 22:16:09 GMT

Hi,

Wondering if this is intentional or not. But I'm trying to set up a "Copy Data" activity in a Pipeline from AWS Redshift. In all of the other data sources I checked there is an option to select a Gateway (exhibit 1). However, when I choose AWS Redshift as a source. There is no option for a gateway (exhibit 2).

Exhibit 1 - Azure SQL Server

Exhibit 2 - Amazon Redshift

Thanks,

Re: Pipeline - Copy Data - Amazon Redshift - No gateway option

Anonymous — Wed, 16 Oct 2024 01:21:12 GMT

Hi coverturfLSH

Unlike some other data sources, such as Azure SQL Server, AWS Redshift does not require a gateway for data transfer. This is primarily because Redshift is designed to work directly with AWS services, particularly Amazon S3, which is often used for data loading.

Direct Connection: Redshift can connect directly to data sources like S3, DynamoDB, or EMR without needing a gateway.

Data Loading: The typical method for loading data into Redshift is through the COPY command, which pulls data from S3 or other supported sources directly into your Redshift tables.

Configuration: Ensure that your IAM roles and permissions are correctly set up to allow Redshift to access the data sources.

Amazon Redshift connector overview - Microsoft Fabric | Microsoft Learn

Configure Amazon Redshift in a copy activity - Microsoft Fabric | Microsoft Learn

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Pipeline - Copy Data - Amazon Redshift - No gateway option

coverturfLSH — Wed, 16 Oct 2024 17:55:19 GMT

Hi Nono,

I may not fully understand the technical side of your response. I'm currently using a gateway that my IT team set up to connect to Redshift. They manage the connection via a service account and granted me access to the gateway. I've been using the gateway to connect to Redshift to refresh my semantic models and dataflows.

When I try to create a copy activity in the pipeline it asks for a username and password which I'm not sure the IT team would be willing to share with me. Any other thoughts on how I could create a connection to Redshift?

Re: Pipeline - Copy Data - Amazon Redshift - No gateway option

Anonymous — Thu, 17 Oct 2024 00:56:27 GMT

Hi coverturfLSH

"However, when I choose AWS Redshift as a source. There is no option for a gateway (exhibit 2)." First of all, in response to your question: The documentation states that it does not require a gateway.

https://learn.microsoft.com/en-us/fabric/data-factory/connector-amazon-redshift-overview#support-in-data-pipelines

This is the answer to your question.

Since your IT team is connected through service account management, you can ask them if they can create a specific role or user for you with the necessary permissions.

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Pipeline - Copy Data - Amazon Redshift - No gateway option

coverturfLSH — Thu, 17 Oct 2024 15:25:54 GMT

Hi Nono,

Thank you for your response. I guess my question is why not offer/use a gateway for the connection. I know it's not required per your earlier reponse. However, it DOES work with a gateway with the DFG2s. Perhaps there's some technical reason that it wants to use a direct connection instead of a gateway.

However it's unexpected behavior that we would need to utilize a different authentication method than we have been using with PowerBI (semantic model and DFG2 refreshes) this entire time.

Re: Pipeline - Copy Data - Amazon Redshift - No gateway option

Anonymous — Fri, 18 Oct 2024 02:28:02 GMT

Hi coverturfLSH

Have you solved the problem now?

Regards,

Nono Chen

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: Pipeline - Copy Data - Amazon Redshift - No gateway option

coverturfLSH — Fri, 18 Oct 2024 16:17:45 GMT

Technically yes, as there does not seem to be a solution for my problem.

Re: Pipeline - Copy Data - Amazon Redshift - No gateway option

FatherLizard — Wed, 12 Mar 2025 18:12:01 GMT

Having the same issue.