Forum Discussion
Dynamic Data Source and Power Query syntax
- 4 years ago
Web.Contents with the api is my preferred method of accessing SharePoint data as you typically get way better performance compared to the OOTB SharePoint connectors and OData.Feed, especially when dealing with libraries/lists with lookups. To answer your questions in order:
Q: ... So, I wanted to know if I am doing anything wrong or that Headers may not be required by doing some other changes?
A: You want to always include the header as you have it to tell the api to return data in JSON format. Otherwise, it returns data in XML format by default, which is much more annoying to work with in Power Query. FYI, probably the reason you get an error when you don't specify Header/accept is because you still have Json.Document in your next step, trying to parse XML, which is resulting in an error.Q: But here, data is returned as 'Records'[?]
A: The way that Json.Document parses the data is to treat it like one giant nested record, which makes sense if you look at JSON syntax, which is basically a bunch of field/value pairings where a value can be a single value, an array of values, or an array of additional field/value pairings. A typical SharePoint list or library items api reponse in JSON would look like:{ **one or two query-specific odata field/values**, "value": [ { **A few item-specific odata field/values**, "Id": 1, "Title": "Title1", "ID": 1 }, { **A few item-specific odata field/values**, "Id": 2, "Title": "Title2", "ID": 2 }, { **A few item-specific odata field/values**, "Id": 3, "Title": "Title3", "ID": 3 } ] }So, as you can see from above, the main data (list items) appear as an array of records in the top-level "value" field.
Thus, if your Source is the Web.Contents wrapped in Json.Document, Source[value] should give you a list of records like the below:Q: ... and I have to expand 3 times[?]
A: Not exactly sure what kind of operations you are doing when you "expand 3 times", but in almost any context, the best method in my experience to handle converting a list of records to a table is Table.FromRecords. Again, if your Source step is the Json.Document/Web.Contents formula, and columns you include in your select argument in the query are not yet finalized, your next step should look something like:= Table.FromRecords( Source[value], null, MissingField.UseNull )And once you have finalized what columns you are querying, I've found it to be a good practice to define your expected table schema, e.g. if your query select is "Id,Title":
= Table.FromRecords( Source[value], type table [Id=nullable number, Title=nullable text], MissingField.UseNull )And before you ask, yes you can specify complex fields when setting the schema. E.g. example with Author (Created By) included where we specified Author/Title, Author/EMail, and Author/JobTitle in the select:
= Table.FromRecords( Source[value], type table [Id=nullable number, Title=nullable text, Author=nullable [Title=text,EMail=text,JobTitle=text] ], MissingField.UseNull )I'll actually split up my complex field definitions, defining before a full table definition, then put it all together in the Table.FromRecords parsing - here is a full M advanced editor example - note the extra work needed for multiselects:
let Source = Json.Document( Web.Contents( "[site url]", [ RelativePath = "_api/web/lists(guid'[list guid]')/items", Headers = [accept="application/json"], Query = [ #"$select"="Id,Title,Created,Author/Title,Author/EMail,Author/JobTitle,MultiSelectLookup/Title", #"$expand"="Author,MultiSelectLookup", #"$top"="5000" ] ] )), AuthorSchema = type nullable [Title=text,EMail=text,JobTitle=text], MultiSelectLookupSchema = type table [Title=nullable text], TableSchema = type table [ Id=nullable Int64.Type, Title=nullable text, Created = nullable text, Author=AuthorSchema, MultiSelectLookup=nullable list ], ParseData = Table.FromRecords( Source[value], TableSchema , MissingField.UseNull ), ParseTableCols = Table.TransformColumns( ParseData, {{ "MultiSelectLookup", each Table.FromRecords(_, MultiSelectLookupSchema, MissingField.UseNull), MultiSelectLookupSchema }} ), TransformTypes = Table.TransformColumnTypes(ParseTableCols,{{"Created", type datetime}}) in TransformTypesAdditional notes:
- As you can see from my above example, I prefer using guid's rather than titles given that titles can change. You can extract the guid from the url of the list settings page in SharePoint, or query _api/web/lists to get all the lists and their attributes including guid (Id field)
- It's not true for some types of queries through the SharePoint api (e.g. querying doc library folder), but definitely when dealing with lists you'll need to factor paging into your query. The api only returns 100 items by default, with a max allowance in one web call of 5000 items if you specify with $top in the query. That means, if you need to return >5000 items, you'll need to do multiple calls.
- Re: setting schema, note that most simple values come in as either text, number, or boolean (and pretty sure nothing else). E.g. dates will come in as text in the 'YYYY-MM-DDTHH:NN:SSZ' format. It's still useful IMO to set the schema because a) saves you trouble of removing the extra ID column that always comes in, b) puts columns in the order you specify, and c) allows you to set the type and only use Table.TransformColumnTypes on columns that actually require transformation (okay, last point only applies for the pedantic among us)
- Multiselect fields usually show up as lists of records, which, as discussed above, Table.FromRecords is most effective at parsing. You'll initialize these as a list column, and then can take an additional step to transform into the correctly typed table column
- In a lot of cases you can actually forgo all the nullable / MissingField.UseNull, but I've found accounting for missing fields makes the query more durable over time.
Edit: minor grammar fixes, trying to fix whitespace, removed application/json;odata=nometadata code snippet I decided not to comment on
Web.Contents with the api is my preferred method of accessing SharePoint data as you typically get way better performance compared to the OOTB SharePoint connectors and OData.Feed, especially when dealing with libraries/lists with lookups. To answer your questions in order:
Q: ... So, I wanted to know if I am doing anything wrong or that Headers may not be required by doing some other changes?
A: You want to always include the header as you have it to tell the api to return data in JSON format. Otherwise, it returns data in XML format by default, which is much more annoying to work with in Power Query. FYI, probably the reason you get an error when you don't specify Header/accept is because you still have Json.Document in your next step, trying to parse XML, which is resulting in an error.
Q: But here, data is returned as 'Records'[?]
A: The way that Json.Document parses the data is to treat it like one giant nested record, which makes sense if you look at JSON syntax, which is basically a bunch of field/value pairings where a value can be a single value, an array of values, or an array of additional field/value pairings. A typical SharePoint list or library items api reponse in JSON would look like:
{
**one or two query-specific odata field/values**,
"value": [
{
**A few item-specific odata field/values**,
"Id": 1,
"Title": "Title1",
"ID": 1
},
{
**A few item-specific odata field/values**,
"Id": 2,
"Title": "Title2",
"ID": 2
},
{
**A few item-specific odata field/values**,
"Id": 3,
"Title": "Title3",
"ID": 3
}
]
}
So, as you can see from above, the main data (list items) appear as an array of records in the top-level "value" field.
Thus, if your Source is the Web.Contents wrapped in Json.Document, Source[value] should give you a list of records like the below:
Q: ... and I have to expand 3 times[?]
A: Not exactly sure what kind of operations you are doing when you "expand 3 times", but in almost any context, the best method in my experience to handle converting a list of records to a table is Table.FromRecords. Again, if your Source step is the Json.Document/Web.Contents formula, and columns you include in your select argument in the query are not yet finalized, your next step should look something like:
= Table.FromRecords( Source[value], null, MissingField.UseNull )
And once you have finalized what columns you are querying, I've found it to be a good practice to define your expected table schema, e.g. if your query select is "Id,Title":
= Table.FromRecords( Source[value], type table [Id=nullable number, Title=nullable text], MissingField.UseNull )
And before you ask, yes you can specify complex fields when setting the schema. E.g. example with Author (Created By) included where we specified Author/Title, Author/EMail, and Author/JobTitle in the select:
= Table.FromRecords( Source[value], type table [Id=nullable number, Title=nullable text, Author=nullable [Title=text,EMail=text,JobTitle=text] ], MissingField.UseNull )
I'll actually split up my complex field definitions, defining before a full table definition, then put it all together in the Table.FromRecords parsing - here is a full M advanced editor example - note the extra work needed for multiselects:
let
Source = Json.Document( Web.Contents(
"[site url]",
[
RelativePath = "_api/web/lists(guid'[list guid]')/items",
Headers = [accept="application/json"],
Query = [
#"$select"="Id,Title,Created,Author/Title,Author/EMail,Author/JobTitle,MultiSelectLookup/Title",
#"$expand"="Author,MultiSelectLookup",
#"$top"="5000"
]
]
)),
AuthorSchema = type nullable [Title=text,EMail=text,JobTitle=text],
MultiSelectLookupSchema = type table [Title=nullable text],
TableSchema = type table [
Id=nullable Int64.Type, Title=nullable text,
Created = nullable text, Author=AuthorSchema,
MultiSelectLookup=nullable list
],
ParseData = Table.FromRecords( Source[value], TableSchema , MissingField.UseNull ),
ParseTableCols = Table.TransformColumns(
ParseData,
{{
"MultiSelectLookup",
each Table.FromRecords(_, MultiSelectLookupSchema, MissingField.UseNull),
MultiSelectLookupSchema
}}
),
TransformTypes = Table.TransformColumnTypes(ParseTableCols,{{"Created", type datetime}})
in
TransformTypes
Additional notes:
- As you can see from my above example, I prefer using guid's rather than titles given that titles can change. You can extract the guid from the url of the list settings page in SharePoint, or query _api/web/lists to get all the lists and their attributes including guid (Id field)
- It's not true for some types of queries through the SharePoint api (e.g. querying doc library folder), but definitely when dealing with lists you'll need to factor paging into your query. The api only returns 100 items by default, with a max allowance in one web call of 5000 items if you specify with $top in the query. That means, if you need to return >5000 items, you'll need to do multiple calls.
- Re: setting schema, note that most simple values come in as either text, number, or boolean (and pretty sure nothing else). E.g. dates will come in as text in the 'YYYY-MM-DDTHH:NN:SSZ' format. It's still useful IMO to set the schema because a) saves you trouble of removing the extra ID column that always comes in, b) puts columns in the order you specify, and c) allows you to set the type and only use Table.TransformColumnTypes on columns that actually require transformation (okay, last point only applies for the pedantic among us)
- Multiselect fields usually show up as lists of records, which, as discussed above, Table.FromRecords is most effective at parsing. You'll initialize these as a list column, and then can take an additional step to transform into the correctly typed table column
- In a lot of cases you can actually forgo all the nullable / MissingField.UseNull, but I've found accounting for missing fields makes the query more durable over time.
Edit: minor grammar fixes, trying to fix whitespace, removed application/json;odata=nometadata code snippet I decided not to comment on
Thank You very much for this!
Really helpled in clarifying the fundamentals..
About the 3 times expansion, I am not sure as well. Previously, I did try the Table.FromRecords, but it gave me a conversion error. (don't remember exactly what now ). So, I did not look too much into it as I was getting my final output of all data correctly after all the expansions so did not bother too much 😅
- MarkLaf4 years agoSuper User
Table.FromRecords with just the first argument usually will work, but I've found with some SharePoint sources (I think it usually comes down to if even just one item in the query somehow got a field blown away), it will only work if you specify MissingField.UseNull. So, unless I'm being lazy/fast, I'll always include the null (because validator yells at you if you try to leave empty - okay for DAX, not M), and MissingField.UseNull in 2nd and 3rd arguments.