Forum Discussion

BW_RFA's avatar
BW_RFA
Helper I
2 years ago

Systematically Look for files.

I ingest data from providers who store files in an SFTP server, one file a day. We do not know when the file will be uploaded. Can you give advice on the best way to systematically try to use the copy activity to ingest the file only once each day, making sure never to lose the data or duplicate the copy?

 

    • Currently, I have the copy activity on a schedule of 15 minutes and using the “filter by last modified” setting I use the dynamic content @addMinutes(utcNow(), -15). However, this does not guarantee I will get the data if the pipeline is delayed for some reason. For example, the pipeline runs on a schedule at 09:00 and 09:15, but the second run was delayed and only started at 09:17. The data file was uploaded at 09:01 and this meant we have not ingested the file on this day.
  •         The only other time function is “triggerTime” but this is not the scheduled trigger time, but the actual time the pipeline started.

Advice Welcome

General Question
pipelines
Tips and Tricks

3 Replies

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi BW_RFA 

     

    Thanks for using Microsoft Fabric Community.

    At this time, we are reaching out to the internal team to get some help on this .
    We will update you once we hear back from them.

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Microsoft Employee
      2 years ago

      Hi BW_RFA 

       

      Apologies for the issue you have been facing and delay in the response from my end. I would like to check are you still facing this issue? 

      If the issue still persists, please reach out to our support team.

      Please go ahead and raise a support ticket to reach our support team:

      https://support.fabric.microsoft.com/support

       

      After creating a Support ticket please provide the ticket number as it would help us to track for more information.

       

      Thank you.

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Microsoft Employee
        2 years ago

        Hi BW_RFA 

         

        We haven’t heard from you on the last response and was just checking back to see if you got a chance to raise a support. If you have raised the support ticket, could you please share us the support ticket number, So that I can reach out to the support team internally for more information. We will update you once we hear back from them.

         

        Thanks.