Forum Discussion
SharePoint list query alternative or optimization
- 5 years ago
hjaf FYI that I finally made a video to describe this approach, and am adding it here for others that may find this post. It also gets the count of items and makes the right number of API calls.
Get SharePoint List Data with Power BI ... Fast - YouTube
Also, a reminder to mark one of these as the solution.
Regards,
Pat
Hi mahoneypat ... I am using your approach to get the items using REST API and it works really fast. But I have published the file to the power bi service and I cannot set the schedule refresh because I have this error: "You can't schedule refresh for this dataset because the following data sources currently don't support refresh".
Do you know how to overcome this problem?
Thanks
Someone in this forum said to look up a solution here: https://blog.crossjoin.co.uk/2016/08/16/using-the-relativepath-and-query-options-with-web-contents-in-power-query-and-power-bi-m-code/
I did so, and after reading a number of posts on the subject I was able to get it working in PowerBI online service. I won't bore you with a lot of details, here's just a simple modified M Code you need to use instead.
What this does is uses a static URL for the first parameter of the Web.Contents() function, then uses a little known (apparently) 2nd parameter to actually pass the rest of the URL (relative path) and query/value pairs into the query string.
You should be able to just modify the bold parts of this and be ready to go. I modified this from the original on the blog post and it even has the original commented-out bits. You can remove those if you want.
I actually hard-coded most of the paths instead of using baseurl or other variables. I probably could have still used them but I wanted immediate success 🙂
This has been an aboslute lifesaver to me. Now my list with 120K list items only takes 15 seconds to refresh.
let
baseurl = "https://TENANT.sharepoint.com/sites/SITE/_api/web/lists/GetByTitle('LIST')/",
itemcount = Json.Document(Web.Contents("https://TENANT.sharepoint.com/sites/SITE/_api/web/lists/GetByTitle('LIST')/"&"ItemCount", [Headers=[Accept="application/json"]]))[value],
skiplist = List.Numbers(0, Number.RoundUp(itemcount/5000), 5000),
#"Converted to Table" = Table.FromList(skiplist, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Renamed Columns" = Table.RenameColumns(#"Converted to Table",{{"Column1", "Skip"}}),
#"Changed Type" = Table.TransformColumnTypes(#"Renamed Columns",{{"Skip", type text}}),
fieldselect = "&$top=5000", // all fields with no expansion
//fieldselect = "&$top=5000&$select = Id,Title,Person,Date", // list desired fields (no expansion)
//fieldselect = "&$top=5000&$select=Id,Title,Choice,LookupColumn/Title,LookupColumn/Project,LookupColumn/ProjectStatus,Date,Person/LastName,Person/FirstName,Person/EMail&$expand=LookupColumn,Person",
//Custom1 = Table.AddColumn(#"Changed Type", "Items", each Json.Document(Web.Contents("baseurl" & "/items?$skipToken=Paged=TRUE%26p_ID=" & [Skip] & fieldselect, [Headers=[Accept="application/json"]]))),
Custom1 = Table.AddColumn(#"Changed Type", "Items", each Json.Document(Web.Contents(
"https://TENANT.sharepoint.com/sites/SITE/_api/web/lists/GetByTitle('LIST')/",
[
RelativePath="/items?$skipToken=Paged=TRUE%26p_ID=" & [Skip] & fieldselect,
Headers=[Accept="application/json"]
]
)
)
),
#"Expanded Items" = Table.ExpandRecordColumn(Custom1, "Items", {"value"}, {"value"}),
#"Expanded value" = Table.ExpandListColumn(#"Expanded Items", "value")
in
#"Expanded value"
- mahoneypat5 years agoMicrosoft Employee
Glad you found that approach. I was about to post a link to same article.
Pat
- lcasey4 years agoPost Prodigy
Hello CmdrKeene,
Where do I paste this code in Power BI? Do I just paste it intop a Blank query?
Thanks
- CmdrKeene4 years agoHelper IV
Hello lcasey,
Yes that's basically the process. Create a new blank query, go to the Advanced Editor from the view toolbar, delete everything and paste in my code instead.
- stevenm154 years agoNew Member
Hi CmdrKeene, first of all, thank you for your code, it worked in Power Bi Service, however I have a question, your code get all the values of the SharePoint List but i want to get also the records (see my picture), do you know how i could do that?
- CmdrKeene4 years agoHelper IV
Great question. For the Author/Editor, those are really just references to another SharePoint table/list called "User Information List" that exists on all SharePoint sites. So for my purposes I retreive this list as well and then setup a relationship in my PowerBI report between the two.
SP Tickets (my main giant list with 200,000 records and growing)
SP UserMeta (just the User Information List with a custom name)
This could be handled in later steps of the query instead but I went this path.
- omelo3 years agoHelper III
Hi CmdrKeene ,
I followed the process described, and I published it to the service. Set the refresh. No issues!
Only to find out that no new data is actually pushed onto the report, not even on the PBI desktop the data refreshed.
I only get new data if I create a new report.I double-check and I have no filters. The item count is accurate, but that is not the number I get on the visuals.
I've cleared the cache but it did not make a difference.Do you have any suggestions to fix this issue?
Thank you for your help!
- CmdrKeene3 years agoHelper IV
Since The item count is accurate, that's not the number I get on the visuals... I think there must be a simple issue like a page or report-wide filter is at play, or perhaps the aggregation is set to count instead of sum (something like that nature).
However I want to also mention something I considered it could be -- this uses the ID number of the sharepoint records. If list entries ever get deleted from your sharepoint list, the ID numbers could be the issue.
- PowerQA13 years agoFrequent Visitor
I am new to trying share point connections which is required since I have to utilize an Enterprise Gateway.
Also I work at a large Corporation. So the share point path goes very deep.
It looks like the only way to get connected to the correct file is to utilize filters.
Refreshing that data set takes a real long time.
So I have 2 questions. Is filtering the way to go or is there another approach.
Would this proposed solution work for my situation?