Forum Discussion
Help Optimize Query: Finding matching Answer in a log of Queries and Answers
Hi,
Being very new and naive to Power Query, I've managed to create a query that works, but that ends up being unacceptably slow. I'm sure there is a much better way, and I would appreciate your guidance !
Context: my source table contains a log of events, where the main columns are:
- [QueryID] - a string than identifies a transaction/query
- [Status] - The type of log entry associated with the Query, one of "Opened" (=request issued) or "Answered" or "Closed" (= responses to the request ... "Closed" is for now equivalent to "Answered").
- [Changed At] - a timestamp
Small complication: within a single [QueryID] there can be more than one request "Opened" and then "Answered".
Goal: I want to extract a list of recorded transactions (opened & answered), along with the time it took for the transaction to be answered.
Step-by-step, I have manged to write the following query which delivers the correct result, but gets inacceptably slow when confronted with large data sets:
= Table.AddColumn(#"Sorted Rows", "End At", each if [Status] <> "Opened" then "--" else
let qq=[QueryID], minT=[Changed At]
in List.Min( Table.Column( Table.SelectRows(#"Sorted Rows", each ( [QueryID]=qq and ([Status]="Answered" or [Status]="Closed") and [Changed At]>=minT ) ), "Changed At") ) )
For all "Opened" message entries, this will find a corresponding "Answered" entry (the closest subsequent entry in chronological order), retrieve its timestamp, and store it in a new colum "End At". This way I will get a list of all the intervals.
Of course this gets really slow for large logs. I can think of ways to optimize this by creating separate intermediate tables and queries. But I strongly suspect there is a much more straightforward way to perform the above operation, which may avoid manual optimization...
Can somebody help, and teach me proper tricks ?
Learn about grouping, especially GroupKind.Local and the fifth parameter.
Obligatory shout out to ImkeF
4 Replies
- v-menakakotaCommunity Support
Hi ivec ,
I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution so other members can easily find it.
Thank you.- v-menakakotaCommunity Support
Hi ivec ,
Thank you lbendlin for providing possible solution.I wanted to check if you had the opportunity to review the information provided. Please feel free to contact us if you have any further questions. If my response has addressed your query, please accept it as a solution and give a 'Kudos' so other members can easily find it.
Thank you.- v-menakakotaCommunity Support
Hi ivec ,
I hope this information is helpful. Please let me know if you have any further questions or if you'd like to discuss this further. If this answers your question, please Accept it as a solution and give it a 'Kudos' so others can find it easily.
Thank you.