Forum Discussion
dynamic data source refresh in Power BI online
I have a report that is working fine in desktop and can be refreshed, but I am not able to refresh the dataset using Power BI Online as it states it has a dynamic data source. As I am only able to pull 8,500 record with each API call and there are usually about 50,000 records, I am using List.Generate and an offset for multiple API calls.
I have the below function that is used to call an API and has an offset that is dynamic to know how much data to pull. The function is below
FNGetSN
= (offset) =>
let
Source = Json.Document(Web.Contents("https://dummyURL.com/api/now/table/u_incident_user?sysparm_query=inc_sys_created_onONLast%203%20months%40javascript%3Ags.beginningOfLast3Months()%40javascript%3Ags.endOfLast3Months()%5Einc_u_customer!%3D5136503cc611227c0183e96598c4f706%5Einc_u_customer!%3Dbf5f738c6f596500c64dda385e3ee421%5Einc_u_customer!%3D6d3a9ebcdbba8300d6fefe9b0c9619ae%5Einc_u_customer!%3Df7a56e28dbe8cc94fbde9ce8db961917%5Einc_u_customer!%3D67035042dbd36b40fbde9ce8db96194c%5Einc_u_customer!%3De68317b66f94b90088d18be54b3ee462%5Einc_u_customer!%3D0e9fb38c6f596500c64dda385e3ee4a0%5Einc_u_customer!%3D3d7832fa4f75e20050062c518110c78d%5Einc_u_customerNOT%20LIKEDeskside&sysparm_display_value=true&sysparm_exclude_reference_link=true&sysparm_fields=inc_number%2Cinc_state%2Cinc_sys_created_on%2Cinc_u_call_type%2Cinc_contact_type%2Cinc_u_customer%2Cusr_email%2Cinc_location%2Cinc_priority%2Cinc_u_business_service%2Cinc_category%2Cinc_short_description%2Cinc_assignment_group%2Cinc_assigned_to%2Cinc_resolved_at%2Cinc_close_notes%2Cinc_calendar_duration&sysparm_limit=8500&sysparm_offset=" & Number.ToText ( offset ))),
#"Converted to Table" = Table.FromRecords({Source}),
#"Expanded result" = Table.ExpandListColumn(#"Converted to Table", "result"),
#"Expanded result1" = Table.ExpandRecordColumn(#"Expanded result", "result", {"inc_u_call_type", "usr_email", "inc_u_customer", "inc_location", "inc_number", "inc_sys_created_on", "inc_close_notes", "inc_category", "inc_contact_type", "inc_state", "inc_priority", "inc_u_business_service", "inc_resolved_at", "inc_assignment_group", "inc_assigned_to", "inc_short_description", "inc_calendar_duration"}, {"result.inc_u_call_type", "result.usr_email", "result.inc_u_customer", "result.inc_location", "result.inc_number", "result.inc_sys_created_on", "result.inc_close_notes", "result.inc_category", "result.inc_contact_type", "result.inc_state", "result.inc_priority", "result.inc_u_business_service", "result.inc_resolved_at", "result.inc_assignment_group", "result.inc_assigned_to", "result.inc_short_description", "result.inc_calendar_duration"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded result1",{{"result.inc_u_call_type", type text}, {"result.usr_email", type text}, {"result.inc_u_customer", type text}, {"result.inc_location", type text}, {"result.inc_number", type text}, {"result.inc_sys_created_on", type datetime}, {"result.inc_close_notes", type text}, {"result.inc_category", type text}, {"result.inc_contact_type", type text}, {"result.inc_state", type text}, {"result.inc_priority", Int64.Type}, {"result.inc_u_business_service", type text}, {"result.inc_resolved_at", type datetime}, {"result.inc_assignment_group", type text}, {"result.inc_assigned_to", type text}, {"result.inc_short_description", type text}, {"result.inc_calendar_duration", type text}})
in
#"Changed Type"
And then I have a query that firstly uses an API to get a count of records and then uses that count to determine how many tables it needs to build. See below
SN (Query)
let
apiResponse = Json.Document(Web.Contents("https://dummyURL.com/api/now/stats/u_incident_user?sysparm_query=inc_sys_created_onONLast%203%20months%40javascript%3Ags.beginningOfLast3Months()%40javascript%3Ags.endOfLast3Months()%5Einc_u_customer!%3D5136503cc611227c0183e96598c4f706%5Einc_u_customer!%3Dbf5f738c6f596500c64dda385e3ee421%5Einc_u_customer!%3D6d3a9ebcdbba8300d6fefe9b0c9619ae%5Einc_u_customer!%3Df7a56e28dbe8cc94fbde9ce8db961917%5Einc_u_customer!%3D67035042dbd36b40fbde9ce8db96194c%5Einc_u_customer!%3De68317b66f94b90088d18be54b3ee462%5Einc_u_customer!%3D0e9fb38c6f596500c64dda385e3ee4a0%5Einc_u_customer!%3D3d7832fa4f75e20050062c518110c78d%5Einc_u_customerNOT%20LIKEDeskside&sysparm_count=true")),
result = apiResponse[result],
stats = result[stats],
count = Number.FromText(stats[count]),
Source = List.Generate(
() => [offset = 8500, SNR = FNGetSN( 0 )],
each [offset] <= count+8500,
each [offset = [offset] + 8500, SNR = FNGetSN([offset])],
each [SNR]
),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandTableColumn(#"Converted to Table", "Column1", {"result.inc_u_call_type", "result.usr_email", "result.inc_u_customer", "result.inc_location", "result.inc_number", "result.inc_sys_created_on", "result.inc_close_notes", "result.inc_category", "result.inc_contact_type", "result.inc_state", "result.inc_priority", "result.inc_u_business_service", "result.inc_resolved_at", "result.inc_assignment_group", "result.inc_assigned_to", "result.inc_short_description", "result.inc_calendar_duration"}, {"result.inc_u_call_type", "result.usr_email", "result.inc_u_customer", "result.inc_location", "result.inc_number", "result.inc_sys_created_on", "result.inc_close_notes", "result.inc_category", "result.inc_contact_type", "result.inc_state", "result.inc_priority", "result.inc_u_business_service", "result.inc_resolved_at", "result.inc_assignment_group", "result.inc_assigned_to", "result.inc_short_description", "result.inc_calendar_duration"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded Column1",{{"result.inc_sys_created_on", type datetime}}),
in
#"Changed Type"
Is there any way around this, as it works exactly as I want, but I just need to be able to refresh the data in Power BI Online?
I would really appreciate any guidance and support on this.
Thanks
Please refer to the documentation. Use RelativePath and Query. Web.Contents - PowerQuery M | Microsoft Learn
1 Reply
- lbendlinSuper User
Please refer to the documentation. Use RelativePath and Query. Web.Contents - PowerQuery M | Microsoft Learn