topic Re: Systematically Look for files. in Pipelines

Systematically Look for files.

BW_RFA — Fri, 02 Feb 2024 15:49:30 GMT

I ingest data from providers who store files in an SFTP server, one file a day. We do not know when the file will be uploaded. Can you give advice on the best way to systematically try to use the copy activity to ingest the file only once each day, making sure never to lose the data or duplicate the copy?

Currently, I have the copy activity on a schedule of 15 minutes and using the “filter by last modified” setting I use the dynamic content @addMinutes(utcNow(), -15). However, this does not guarantee I will get the data if the pipeline is delayed for some reason. For example, the pipeline runs on a schedule at 09:00 and 09:15, but the second run was delayed and only started at 09:17. The data file was uploaded at 09:01 and this meant we have not ingested the file on this day.

The only other time function is “triggerTime” but this is not the scheduled trigger time, but the actual time the pipeline started.

Advice Welcome

Re: Systematically Look for files.

v-cboorla-msft — Mon, 05 Feb 2024 10:23:14 GMT

Hi BW_RFA

Thanks for using Microsoft Fabric Community.

At this time, we are reaching out to the internal team to get some help on this .
We will update you once we hear back from them.

Re: Systematically Look for files.

v-cboorla-msft — Mon, 19 Feb 2024 04:51:46 GMT

Hi BW_RFA

Apologies for the issue you have been facing and delay in the response from my end. I would like to check are you still facing this issue?

If the issue still persists, please reach out to our support team.

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thank you.

Re: Systematically Look for files.

v-cboorla-msft — Tue, 20 Feb 2024 05:52:58 GMT

Hi BW_RFA

We haven’t heard from you on the last response and was just checking back to see if you got a chance to raise a support. If you have raised the support ticket, could you please share us the support ticket number, So that I can reach out to the support team internally for more information. We will update you once we hear back from them.

Thanks.