Starting December 3, join live sessions with database experts and the Microsoft product team to learn just how easy it is to get started
Learn moreGet certified in Microsoft Fabric—for free! For a limited time, get a free DP-600 exam voucher to use by the end of 2024. Register now
Good Day,
I am using power query to automatically pull data from 3 source files every 5 minutes. These source files automatically update at different frequencies from our ERP system . When the power query performs an auto update the same time the source file is in the process of being updated, I get the following error:
Upon acknowledgement and subsequent refresh, it works as desired.
It would be greatly appreciated if I can get help - an ideal solution would automatically close the warning prompt and display the same information until power query performs the next auto update. Any straight forward VBA code can help with this?
Much appreciated.
Thanks for the reply. Believe through Phython it pulls the data to a local network folder from the ERP and saves over the previous file. Was hoping an error handling VBA would be able to acknowdlege and subsequently ignore the errors until the next power query refresh iteration.
The issue with this is the way in which Power Query works. It wipes everything before it starts attempting the new import and transformation process. Therefore, as soon as you start, you have generally to complete or you get a failure with no original data retention.
If the refresh frequency weren't so high I'd suggest putting these into Dataflows, which would actually retain your original data upon failure, but then you're chaining two refresh processes into a narrow five-minute window each time, so risks refresh overlap. If these are very small files and refresh in a matter of seconds, this would be the way I would go, using Power Automate to detect the completion of a Dataflow refresh and automatically initiate the dataset refresh.
Beyond this, I think you'd want to explore what @jbwtp has suggested, by looping attempts within the query until you get a successful response.
Pete
Proud to be a Datanaut!
Hi @Jeffbruin,
I believe that the problem that you have is because the PQ tries to fetch data while the file is being re-written. Do you mind trying to apply to your code the following pattern:
let
fetch_data = (MaxAttempts, DelayBetweenAttempts) =>
let
Numbers = List.Numbers(1, MaxAttempts),
FileSystemCalls = List.Transform(Numbers, each try Function.InvokeAfter(()=>internal_fetch_data(), if _ > 1 then DelayBetweenAttempts else #duration(0,0,0,0)) otherwise null),
OnlySuccessful = List.Select(FileSystemCalls, each _ <> null),
Result = List.First(OnlySuccessful, null)
in
Result,
internal_fetch_data = () =>
let
// this is your origianl query from the first line to the step that throughs out the error (inclusive on both sides)
Source = Table.FromRows(Json.Document(Binary.Decompress(Binary.FromText("i45W8k0tSk9NUYqNBQA=", BinaryEncoding.Base64), Compression.Deflate)), let _t = ((type nullable text) meta [Serialized.Text = true]) in type table [Merged = _t]),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Merged", type text}})
in #"Changed Type",
#"Uppercased Text" = Table.TransformColumns(fetch_data(10, #duration(0,0,0,10) /*10 sec delay*/),{{"Merged", Text.Upper, type text}})
in
#"Uppercased Text"
This tries to get the data from the file several times each time with some delay. Maybe this will be sufficient in your case and resolve the problem? Obvoiusly, you will nbeed to adjust it to your case by altering the content of the internal_fetch_data function.
Thanks,
John
Hi @Jeffbruin ,
Where does your ERP save the source files to? Would it be possible to use Power Automate to detect a change in file modification date then copy the file somewhere else to be used for Power Query?
Pete
Proud to be a Datanaut!
Starting December 3, join live sessions with database experts and the Fabric product team to learn just how easy it is to get started.
March 31 - April 2, 2025, in Las Vegas, Nevada. Use code MSCUST for a $150 discount! Early Bird pricing ends December 9th.
User | Count |
---|---|
24 | |
13 | |
12 | |
11 | |
8 |
User | Count |
---|---|
43 | |
26 | |
16 | |
15 | |
12 |