Forum Discussion
how to create a query that paginates?
Hi tempranello,
thanks for getting back and providing such a nice code.
Nothing really obvious in the List.Generate part, although: What about "TimeEntries" and "TimeEntry" - cannot see where you use them there and you don't return them in the output. So could you omit them? (Although I wouldn't believe that this is a performance killer).
The next step to localize the performance-problem would be to skip all steps after #"Expanded ImportedXML" This seems to be the step where the first "real" value from the procedure is shown and before further transformations take place.
You can try to put a buffer here:
ConvertToTable = Table.FromList(List.Buffer(DataList), Splitter.SplitByNothing(), null, null, ExtraValues.Error),
making sure to avoid multiple web-calls.
Or you use Fiddler to check how long the web-calls actually take.
Hi ImkeF,
I'm trying to link to a dictionary provided by Investopedia.com with a slicer of 26 English letters plus a "#" mark. Under each option there are hundreds of terms available with a hyperlink to blow by blow explanations correspondingly. In order to reiterate the multiple pages over the web data it seems to require two parameters for alphabet and page number respectively in M language. I put a pbix here for your reference. Hopefully you would like to guide me to complete it. Thanks.
Julian Chen
- ImkeF8 years agoCommunity Champion
Have you refreshed the query? I've used a filter which I might have forgotten to take out.
For learning purposes I'd suggest that you step through the queries: Pretty sure you find the filters (just delete them) and the answer to your URL question in there ;)
- ImkeF8 years agoCommunity Champion
Hi julianchen,
please have a look at this file: https://1drv.ms/u/s!Av_aAl3fXRbehbJKWOtDP86f5R726A
Does the table "Parameter Table_Rows" contain what you're after?
- julianchen8 years agoAdvocate I
Hi ImkeF,
It's amazing you've solved my major issues in a few seconds with only one tiny problem left as follows:
There are 67 terms with non-alphbet prefix missing. Could you please add them back and assign a "#" index to them for slicer selection?
Julian Chen
- ImkeF8 years agoCommunity Champion
Yes, it was late yesterday, so I didn't check it. Please have a look at it again:
1) the "#" is actually a "1" in the URL, so I've added a "TechnicalIndex"-column to your table
2) Also the URL had to be adjusted for all categories with just 1 page, as they don't use a page number as a URL parameter
Hope this is what you need now?
- julianchen8 years agoAdvocate I
Hi ImkeF,
I found the query Parameter Table_Rows just returned 2248 rows covering TechnicalIndex 1, A and B only. The total number is supposed to be 15,937 indeed. Could you please check it again to retrieve all the rows as a result? Sorry for bothering you so much.
By the way, for those categories with just 1 page, it's still valid to include the page number = 0 (like https://www.investopedia.com/terms/a/?page=0) as a URL parameter.
Julian
- julianchen8 years agoAdvocate I
After removing the filter as you alerted, I also modified the Parameter Table_Rows query as shown below (starting from page = 0, and taking out if clause. Finally, I got 19,561 rows - excess 24 records of gap. I will check technicalIndex one after one to dig out what's the gap if possible. That's the learning purposes as you guided me. Thanks again for your great help. I've been expecting to have this "homework" to be done for quite a long time.
let
Source = #"Parameter Table",
#"Added Custom" = Table.AddColumn(Source, "Pages", each {0..[Page]}),
#"Expanded Pages" = Table.ExpandListColumn(#"Added Custom", "Pages"),
#"Added Custom1" = Table.AddColumn(#"Expanded Pages", "Custom", each fnGetPages([TechnicalIndex], Text.From([Pages]))),
// #"Added Custom1" = Table.AddColumn(#"Expanded Pages", "Custom", eachif [Pages] = 1 then fnGetPages([TechnicalIndex], null) elsefnGetPages([TechnicalIndex], Text.From([Pages]))),
#"Removed Errors" = Table.RemoveRowsWithErrors(#"Added Custom1", {"Custom"}),
#"Expanded Custom" = Table.ExpandTableColumn(#"Removed Errors", "Custom", {"Column1", "Term", "Link", "Index"}, {"Column1", "Term", "Link", "Index.1"}),
#"Inserted Text After Delimiter" = Table.AddColumn(#"Expanded Custom", "Text After Delimiter", each Text.AfterDelimiter([Term], "/", 2), type text),
#"Renamed Columns" = Table.RenameColumns(#"Inserted Text After Delimiter",{{"Term", "Term_1"}, {"Text After Delimiter", "Term"}})
in
#"Renamed Columns" - Anonymous8 years agoNot applicable
Hi all,
I have been struggling to paginate JIRA's rest API where the max results can be obtained for only 1000 rows. The pagination technique in this page so far provides a custom code which is really easy in an ideal scenario for a given scenario but when there is no ideal scenario or use different source with custom code which was my scenario and I stumbled on a very easy and an intiutive manner in which it was explained using Invoke Function out of the box instead of creating custom codes. Please refer to this page for more information and this helped to resolve the issue.
https://stackoverflow.com/questions/46904641/play-with-paginated-data-with-power-bi
Kind regards,
A!