Reply
MRosk95
Frequent Visitor

Access to Resource Forbidden - Web Contents

Hi there,

 

Until today, I have been accessing a public website through Power Query and pulling content down. All of a sudden I am getting an "Access to Resource is Forbidden" error.

This is my connection code:

let
// Base URL
baseUrl = "https://www.tenders.act.gov.au",

// Query Parameters
Source = Web.Contents(
baseUrl,
[
RelativePath = "/tender/search",
Query = [
#"keywords" = "",
#"tenderCode" = "",
#"tenderTitle" = "",
#"tenderState" = "OPEN",
#"tenderType" = "",
#"prequalificationCategory" = "",
#"procurementCategory" = "",
#"categories[0].fullTitle" = "43000000 Information Technology Broadcasting and Telecommunications",
#"categories[0].id" = "104789",
#"categories[1].fullTitle" = "Components for information technology or broadcasting or telecommunications",
#"categories[1].id" = "107731",
#"categories[2].fullTitle" = "81160000 Information Technology Service Delivery",
#"categories[2].id" = "145204",
#"issuingBusinessId" = "",
#"awardedSupplier.id" = "",
#"awardedSupplier.name" = "",
#"openThisWeek" = "false",
#"openingDateFrom" = "",
#"openingDateTo" = "",
#"ageRestriction" = "",
#"closeThisWeek" = "false",
#"closingDateFrom" = "",
#"closingDateTo" = "",
#"awardedThisWeek" = "false",
#"awardedDateFrom" = "",
#"awardedDateTo" = "",
#"orderBy" = "",
#"groupBy" = "NONE",
#"page" = "",
#"searchTitle" = ""
],
Headers = [
Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
#"User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36"
]
]
),

 

The website url is: 

https://www.tenders.act.gov.au/tender/search?keywords=&tenderCode=&tenderTitle=&tenderState=OPEN&ten... 

4 REPLIES 4
lbendlin
Super User
Super User

Web.Contents only gets you that far. You need to use Web.BrowserContents and then parse the HTML.

 

You can omit all the query parameters that are blank.

 

let
    Source = Web.BrowserContents("https://www.tenders.act.gov.au/tender/search?tenderState=OPEN&categories%5B0%5D.fullTitle=43000000+Information+Technology+Broadcasting+and+Telecommunications&categories%5B0%5D.id=104789&categories%5B0%5D.percentage=100&openThisWeek=false&closeThisWeek=false&awardedThisWeek=false&groupBy=NONE"),
    #"Extracted Table From Html" = Html.Table(Source, {{"Column1", ".tender-code-state"}, {"Column2", ".tender-code-state .tablesaw-cell-content"}, {"Column3", ".tablesaw-cell-content B"}, {"Column4", ".tender-code-state + *"}, {"Column5", ".strong"}, {"Column6", ".tender-date"}, {"Column7", ".opening_date"}, {"Column8", ".closing_date"}, {"Column9", "DIV:nth-child(2) .line-item-detail"}, {"Column10", "DIV:nth-child(3) .line-item-detail:nth-child(1)"}, {"Column11", ".tender-code-state .tablesaw-cell-label"}, {"Column12", ".tender-row-state"}, {"Column13", ".tender-date .tablesaw-cell-label"}, {"Column14", ".date_label_list:nth-child(1)"}, {"Column15", ".date_label_list:nth-child(5)"}, {"Column16", ".line-item-detail:nth-child(2)"}, {"Column17", ".line-item-detail:nth-child(3)"}}, [RowSelector="TBODY TR"])
in
    #"Extracted Table From Html"

Hey @lbendlin thank you so much for your response.

 

The issue that I continuall come across with Web.BrowserContents() is that I can't get it to refresh in Power Bi Service without installing a on-premises personal gateway? 

 

I could be wrong but why do I need an on-premises gateway to hit a public website? as such Web.Contents is the only option I can seem to use to achieve the outcome I want.

 

If you have a workaround or any ideas it would be a huge help.

 

Thanks,

What you are doing is basically screen scraping. This is frowned upon. The "correct" way would be to use the data API that your data source should provide.  That's what Web.Contents is designed for - for APIs, not for webpage scraping.

Thanks @lbendlin, no API is available and I don't believe the datasource provider is interested in providing one so kind of stuck here but thank you for your help.

avatar user

Helpful resources

Announcements
March PBI video - carousel

Power BI Monthly Update - March 2025

Check out the March 2025 Power BI update to learn about new features.

March2025 Carousel

Fabric Community Update - March 2025

Find out what's new and trending in the Fabric community.

Top Solution Authors (Last Month)
Top Kudoed Authors (Last Month)