Handling Power query refresh errors resulting from source file refresh

Hi Jeffbruin,

I believe that the problem that you have is because the PQ tries to fetch data while the file is being re-written. Do you mind trying to apply to your code the following pattern:

let
    fetch_data = (MaxAttempts, DelayBetweenAttempts) =>
        let
            Numbers = List.Numbers(1, MaxAttempts),
            FileSystemCalls = List.Transform(Numbers, each try Function.InvokeAfter(()=>internal_fetch_data(), if _ > 1 then DelayBetweenAttempts else #duration(0,0,0,0)) otherwise null),
            OnlySuccessful = List.Select(FileSystemCalls, each _ <> null),
            Result = List.First(OnlySuccessful, null)
            in
        Result,


    internal_fetch_data = () =>
        let 
            // this is your origianl query from the first line to the step that throughs out the error (inclusive on both sides)
            Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45W8k0tSk9NUYqNBQA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Merged = _t]),
            #"Changed Type" = Table.TransformColumnTypes(Source,{{"Merged", type text}})
        in #"Changed Type", 
    
    #"Uppercased Text" = Table.TransformColumns(fetch_data(10, #duration(0,0,0,10) /*10 sec delay*/),{{"Merged", Text.Upper, type text}})
in
    #"Uppercased Text"

This tries to get the data from the file several times each time with some delay. Maybe this will be sufficient in your case and resolve the problem? Obvoiusly, you will nbeed to adjust it to your case by altering the content of the internal_fetch_data function.

Thanks,

John 

The issue with this is the way in which Power Query works. It wipes everything before it starts attempting the new import and transformation process. Therefore, as soon as you start, you have generally to complete or you get a failure with no original data retention.

If the refresh frequency weren't so high I'd suggest putting these into Dataflows, which would actually retain your original data upon failure, but then you're chaining two refresh processes into a narrow five-minute window each time, so risks refresh overlap. If these are very small files and refresh in a matter of seconds, this would be the way I would go, using Power Automate to detect the completion of a Dataflow refresh and automatically initiate the dataset refresh.

Beyond this, I think you'd want to explore what jbwtp has suggested, by looping attempts within the query until you get a successful response.

Pete