topic Re: Data Pipeline FTP Fabric in Pipelines

Data Pipeline FTP Fabric

Idonica — Mon, 08 Jan 2024 12:58:06 GMT

Hello everyone,

So my company is starting with Fabric with a POC and on the bronze layer i have encountered issues/limitations that i think it is only solvable only with using notebooks:

I have created a data Pipeline with an FTP connection do get data from a server: 51.133.156.1 (it is not the real ip server).

I have access to it through azure portal as a linked service

But when i try to reach it out the following are happening i don't how to deal:

1) I cannot use the Keyvault, the only option i found doable is by using python notebooks, is this as it is or it will change ? I'm pretty sure Microsoft will not drop the usage of keyvault in fabric

1.1 should is use linked services and blob's to connect in fabric? this defeats the porpose of unifying all under the same wing.

2) i do not know even how to reach 51.133.156.1, because it asks for the server as a website.

2.1) I have seen that, since the server is private and not public, fabric data pipeline/ADF cannot reach it, is that true ?

Thank you in advance,

Re: Data Pipeline FTP Fabric

Anonymous — Tue, 09 Jan 2024 11:59:50 GMT

Hi Idonica ,

Thanks for using Fabric Community.

Currently Keyvault is not supported in Fabric Data Factory. We can expect this in future.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Inorder to guide you with FTP connection I would like to understand what authentication method does your FTP have? Ideally how do you connect to your FTP endpoint?

Hope this helps. Please let me know if you have any further queries.

Re: Data Pipeline FTP Fabric

Idonica — Tue, 09 Jan 2024 14:06:29 GMT

Hi Anonymous ,
Thank you for taking the time to assist me,

So on on the ftp side this is what im doing:

1) 2023 is the port

2) the connection is basic auth

3) user name and pass that im using are from keyvault and when i test them on the azure side as a linked services it is sucesfully connect to the FTP.

Hope this is enough,

Re: Data Pipeline FTP Fabric

Anonymous — Tue, 09 Jan 2024 14:11:27 GMT

Hi Idonica ,

Thanks for the response. So you were to connect using Azure Data Factory - FTP Linked Service, but facing issue or unable to connect while using Fabric Data Factory - Copy Activity.

Re: Data Pipeline FTP Fabric

Idonica — Tue, 09 Jan 2024 14:40:39 GMT

Hi Anonymous,
Not quite,

What i tried to do is connect directly to the FTP using an user that i know that can reach the FTP, which in this case is the user that is in the Linked service.

But if that doesn't work (because as you mentioned, ADF fabric does no support), i might try connecting using linked service+BLOB.

But before, is there any workaround to be able to use keyvault or not yet ?

Re: Data Pipeline FTP Fabric

Anonymous — Tue, 09 Jan 2024 14:50:23 GMT

Hi Idonica ,

At present we don't have any option to use keyvault in Fabric.
At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Re: Data Pipeline FTP Fabric

Idonica — Wed, 10 Jan 2024 10:57:16 GMT

HI Anonymous,
Thank you fior your assistance,

I will wait on the internal team response.

Thanks,

Re: Data Pipeline FTP Fabric

Anonymous — Thu, 11 Jan 2024 10:22:43 GMT

Hi Idonica ,

Fabric currently does not support KeyVault. Until that's fully integrated, there's a simple workaround. As long as you are executing notebook has access to the vault, the below will work.

from trident_token_library_wrapper import PyTridentTokenLibrary as tl access_token = mssparkutils.credentials.getToken("keyvault") tl.get_secret_with_token("https://vaultName.vault.azure.net/","secretName",access_token)

You can also refer this link, for more information.

Incase if Notebooks work aorund doesn't handle your case.

Appreciate if you could share the feedback on our feedback channel. Which would be open for the user community to upvote & comment on. This allows our product teams to effectively prioritize your request against our existing feature backlog and gives insight into the potential impact of implementing the suggested feature.

Hope this is helpful. Please let me know if you have further queries.

Re: Data Pipeline FTP Fabric

Anonymous — Fri, 12 Jan 2024 11:27:39 GMT

Hello Idonica ,

We haven’t heard from you on the last response and was just checking back to see if your query was answered.
Otherwise, will respond back with the more details and we will try to help .

Re: Data Pipeline FTP Fabric

Idonica — Fri, 12 Jan 2024 11:33:38 GMT

Hi Anonymous, thank you for your assist,
I will try to use notebooks, even though i wanted to avoid using notebooks on the Bronze layer.

In the meantime i will mark this as accepted solution!

Have an excelent weekend,

Re: Data Pipeline FTP Fabric

Anonymous — Fri, 12 Jan 2024 11:38:00 GMT

Hi Idonica ,

Glad to know that you got some insights over your query. Please continue using Fabric Community for your further queries.

Thank you.