Forum Discussion
Issue with ‘Scheduled Refresh’ for api calls
- Anonymous5 years ago
Hi Anonymous,
It seems like the power query not allow you direct do merge operations between different types(number and text), please try to use the following code of 'add custom column' step, I added text type convert in the formula.
#"Added Custom" = Table.AddColumn(#"Converted to Table1", "Source", each Json.Document(Web.Contents("https://api.xyz.com/", [ Headers=[#"Gen-Api-Key"="...."], RelativePath="natgas/events/v1/notices/details", Query= [ ids=Text.From([Column1]), format="json" ] ])))Regards,
Xiaoxin Sheng
- Anonymous5 years ago
Hi Anonymous and all: Good day! I found checking the data in the data base table and Api portal that for some pipes there will be same event posted for different locations [as they are impacted for all those locations for a pipeline]. That's why the multiple location rows are coming for those pipes! So, it's actually not an error/issue, but a fact! So, I am closing this thread. Thanks all for your patience. 🙂
Hi Anonymous,
What error that displayed in the custom column who invokes the web connector? Can you please share the detailed error messages?
After I checked your formula and snapshot, I found your id field values are stored in 'column1', so you need to use 'column1' to replace noticed. (you can also try to add 'relativepath' and other optional parameters into the web connector):
#"Added Custom" = Table.AddColumn(#"Converted to Table1", "Source", each Json.Document(Web.Contents("https://api.xyz.com/",
[
Headers=[#"Gen-Api-Key"="...."],
RelativePath="natgas/events/v1/notices/details",
Query=
[
ids=[Column1],
format="json"
]
])))
Using The RelativePath And Query Options With Web.Contents() In Power Query And Power BI M Code
Regards,
Xiaoxin Sheng
Hi Anonymous and all : Good day! Thanks for your continued support. I got the following error:
After I replace NoticeId with Column1, get the following error:
So, I started again and didn’t delete any steps. I renamed “Column1” as “NoticeId”, edited the #“Added Custom” part as below with Relative path and Query, created a variable, id to use in ids=[id]:
-------------------------
let
Source = Json.Document(Web.Contents("https://api.genscape.com/",
[
RelativePath="natgas/events/v1/notices",
Query=
[
priorities="critical",
limit="50",
offset="0",
format="json"
]
,
Headers=[
#"Gen-Api-Key"="f94ffe41c1d74066b90c564397edf9b2"
]
]
)),
data = Source[data],
#"Converted to Table" = Table.FromList(data, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"id", "pipelineName", "priority", "type", "subject", "postDate", "effDate", "endDate"}, {"id", "pipelineName", "priority", "type", "subject", "postDate", "effDate", "endDate"}),
id1 = #"Expanded Column1"[id],
#"Converted to Table1" = Table.FromList(id1, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Added Custom" = Table.AddColumn(#"Converted to Table1", "Custom", each (NoticeId as number) => let
Source = Json.Document(Web.Contents("https://api.genscape.com/",
[
Headers=[#"Gen-Api-Key"="f94ffe41c1d74066b90c564397edf9b2"],
RelativePath=" natgas/events/v1/notices/details",
#"id"= Number.ToText(NoticeId),
Query=
[
ids=[id],
format="json"
]
])),
#"Converted to Table" = Record.ToTable(Source),
#"Expanded Value" = Table.ExpandListColumn(#"Converted to Table", "Value"),
#"Expanded Value1" = Table.ExpandRecordColumn(#"Expanded Value", "Value", {"id", "pipelineName", "pipelineId", "priorNoticeIdentifierId", "noticeIdentifierId", "priority", "type", "status", "subject", "body", "postDate", "effDate", "endDate", "insertDate"}, {"id", "pipelineName", "pipelineId", "priorNoticeIdentifierId", "noticeIdentifierId", "priority", "type", "status", "subject", "body", "postDate", "effDate", "endDate", "insertDate"})
in
#"Expanded Value1")
in
#"Added Custom"
--------------------------
Now I get the following with ‘function’ in each cell under ‘Custom’ Column
Not sure what to do?! Many thanks in advance.
- Anonymous5 years agoNot applicable
Hi Anonymous,
It seems like the power query not allow you direct do merge operations between different types(number and text), please try to use the following code of 'add custom column' step, I added text type convert in the formula.
#"Added Custom" = Table.AddColumn(#"Converted to Table1", "Source", each Json.Document(Web.Contents("https://api.xyz.com/", [ Headers=[#"Gen-Api-Key"="...."], RelativePath="natgas/events/v1/notices/details", Query= [ ids=Text.From([Column1]), format="json" ] ])))Regards,
Xiaoxin Sheng
- Anonymous5 years agoNot applicable
Hi Anonymous: Thanks a lot for your help. It removed the error & I can now pull the desired data. I will check with the Publish team to find out if this code change will resolve the issue/error we have been getting so far reg 'Scheduled Refresh' with dynamic query!
- Anonymous5 years agoNot applicable
Hi Anonymous: Good day. Though the Publish team is not getting the error before, I see an issue in the data. I get a Column, 'locations' as a List, which earlier [before the M code change]- I could see as : delimited with location id & location name. I could then select the column, right click and select 'Split by delimiter'. I just posted the 'split by delimiter' option not showing up in
https://community.powerbi.com/t5/Desktop/Split-a-Single-Column-with-delimiter-colon/td-p/1428948
If I extract as new rows/ expand-> it doesn't work: rather creates multiple rows with same 'id' and shows data incorrectly in a Map visual.
I am thinking is it something more I have to do in the M code to split the list by colon [:] delimiter?
The code for this data source, #2 details_maintenance_events is as below [changing some value with xyz/abc/...]:
---------------
let
Source = Sql.Database("xyz\Fundamentals", "NatGas_V3", [Query="select * from NatGas_V3.natgas.pipeline_maintenance_events#(lf)where endDate >=" &datefilter]),
#"Sorted Rows" = Table.Sort(Source,{{"endDate", Order.Ascending}, {"updateDate", Order.Descending}}),
#"Filtered Rows" = Table.SelectRows(#"Sorted Rows", each true),
id1 = #"Filtered Rows"[id],
#"Converted to Table" = Table.FromList(id1, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Added Custom" = Table.AddColumn(#"Converted to Table", "Source", each Json.Document(Web.Contents("https://api.abc.com/",
[
Headers=[#"Gen-Api-Key"="........"],
RelativePath="natgas/events/v1/maintenance-events/details",
Query=
[
ids=Text.From([Column1]),
format="json"
]
]))),
#"Removed Errors" = Table.RemoveRowsWithErrors(#"Added Custom", {"Column1"}),
#"Removed Errors1" = Table.RemoveRowsWithErrors(#"Removed Errors", {"Column1"}),
#"Removed Errors2" = Table.RemoveRowsWithErrors(#"Removed Errors1", {"Source"}),
#"Expanded Source" = Table.ExpandRecordColumn(#"Removed Errors2", "Source", {"data"}, {"data"}),
#"Expanded data" = Table.ExpandListColumn(#"Expanded Source", "data")
in
#"Expanded data"---------------------
First I pull data from a sql database as #1 maintenance_events data source, then I create this 2nd data source, details_maintenance_events. I haven't changed any code in #1 data source[ use get data-> sql database->....], which is as below:
----
let
Source = Sql.Database("xyz\Fundamentals", "NatGas_V3", [Query="select * from NatGas_V3.natgas.pipeline_maintenance_events#(lf)where endDate >=" &datefilter]),
#"Sorted Rows" = Table.Sort(Source,{{"endDate", Order.Ascending}, {"updateDate", Order.Descending}}),
#"Filtered Rows" = Table.SelectRows(#"Sorted Rows", each true)
in
#"Filtered Rows"--------
Many thanks in advance.
- Anonymous5 years agoNot applicable
Hi Anonymous and all: Good day! I did some code change and posted this in another thread:
https://community.powerbi.com/t5/Desktop/Split-a-Single-Column-with-delimiter-colon/m-p/1430399
Please check my last post in that thread as that's where I am now! Many thanks in advance.
- Anonymous5 years agoNot applicable
Hi Anonymous : The M code change helped the 'Scheduled Refresh' to be activated! Thanks a lot for your support through this process! 🙂
- Anonymous5 years agoNot applicable
Hi Anonymous and all: Good day! I found checking the data in the data base table and Api portal that for some pipes there will be same event posted for different locations [as they are impacted for all those locations for a pipeline]. That's why the multiple location rows are coming for those pipes! So, it's actually not an error/issue, but a fact! So, I am closing this thread. Thanks all for your patience. 🙂