Access to Resource Forbidden - Web Contents

[MRosk95]

Hi there,

Until today, I have been accessing a public website through Power Query and pulling content down. All of a sudden I am getting an "Access to Resource is Forbidden" error.

This is my connection code:

let
// Base URL
baseUrl = "https://www.tenders.act.gov.au",

// Query Parameters
Source = Web.Contents(
baseUrl,
[
RelativePath = "/tender/search",
Query = [
#"keywords" = "",
#"tenderCode" = "",
#"tenderTitle" = "",
#"tenderState" = "OPEN",
#"tenderType" = "",
#"prequalificationCategory" = "",
#"procurementCategory" = "",
#"categories[0].fullTitle" = "43000000 Information Technology Broadcasting and Telecommunications",
#"categories[0].id" = "104789",
#"categories[1].fullTitle" = "Components for information technology or broadcasting or telecommunications",
#"categories[1].id" = "107731",
#"categories[2].fullTitle" = "81160000 Information Technology Service Delivery",
#"categories[2].id" = "145204",
#"issuingBusinessId" = "",
#"awardedSupplier.id" = "",
#"awardedSupplier.name" = "",
#"openThisWeek" = "false",
#"openingDateFrom" = "",
#"openingDateTo" = "",
#"ageRestriction" = "",
#"closeThisWeek" = "false",
#"closingDateFrom" = "",
#"closingDateTo" = "",
#"awardedThisWeek" = "false",
#"awardedDateFrom" = "",
#"awardedDateTo" = "",
#"orderBy" = "",
#"groupBy" = "NONE",
#"page" = "",
#"searchTitle" = ""
],
Headers = [
Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
#"User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
]
]
),

The website url is: 

https://www.tenders.act.gov.au/tender/search?keywords=&tenderCode=&tenderTitle=&tenderState=OPEN&ten... 

[lbendlin]

Web.Contents only gets you that far. You need to use Web.BrowserContents and then parse the HTML.

You can omit all the query parameters that are blank.

let
    Source = Web.BrowserContents("https://www.tenders.act.gov.au/tender/search?tenderState=OPEN&categories%5B0%5D.fullTitle=43000000+Information+Technology+Broadcasting+and+Telecommunications&categories%5B0%5D.id=104789&categories%5B0%5D.percentage=100&openThisWeek=false&closeThisWeek=false&awardedThisWeek=false&groupBy=NONE"),
    #"Extracted Table From Html" = Html.Table(Source, {{"Column1", ".tender-code-state"}, {"Column2", ".tender-code-state .tablesaw-cell-content"}, {"Column3", ".tablesaw-cell-content B"}, {"Column4", ".tender-code-state + *"}, {"Column5", ".strong"}, {"Column6", ".tender-date"}, {"Column7", ".opening_date"}, {"Column8", ".closing_date"}, {"Column9", "DIV:nth-child(2) .line-item-detail"}, {"Column10", "DIV:nth-child(3) .line-item-detail:nth-child(1)"}, {"Column11", ".tender-code-state .tablesaw-cell-label"}, {"Column12", ".tender-row-state"}, {"Column13", ".tender-date .tablesaw-cell-label"}, {"Column14", ".date_label_list:nth-child(1)"}, {"Column15", ".date_label_list:nth-child(5)"}, {"Column16", ".line-item-detail:nth-child(2)"}, {"Column17", ".line-item-detail:nth-child(3)"}}, [RowSelector="TBODY TR"])
in
    #"Extracted Table From Html"

[MRosk95]

Hey @lbendlin thank you so much for your response.

The issue that I continuall come across with Web.BrowserContents() is that I can't get it to refresh in Power Bi Service without installing a on-premises personal gateway? 

I could be wrong but why do I need an on-premises gateway to hit a public website? as such Web.Contents is the only option I can seem to use to achieve the outcome I want.

If you have a workaround or any ideas it would be a huge help.

Thanks,

M 

[lbendlin]

What you are doing is basically screen scraping. This is frowned upon. The "correct" way would be to use the data API that your data source should provide.  That's what Web.Contents is designed for - for APIs, not for webpage scraping.

[MRosk95]

Thanks @lbendlin, no API is available and I don't believe the datasource provider is interested in providing one so kind of stuck here but thank you for your help.