Get / Post Method returns <META NAME="robots" CONTENT="noindex,nofollow">

[kkapsokavadis]

Hi Community,

I need help in order to get results from an API.

With Postman everythings works fine, but with power query i don't have the desired result (most times)

I have made the POST request based on all the guidelines that i read through internet but the output after webcontents call anf after converting to text is the below

 if picture is not very clear, always 3rd line has the follwing text
<META NAME="robots" CONTENT="noindex,nofollow">

and 4th line 
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">

Sometimes i have managed to get the real response (which is an html response) by passing in Headers the whole Cookie (cookie from postman or cookie from browser) but i have realised that after few calls, then the above message is returned and not the real response.

Does anyone have an idea hot to overcome this problem??

[PhilipTreacy · Accepted solution]

Hi @kkapsokavadis 

The website you're trying to access is being protected by a web firewall from Imperva/Incapsula and your requests are being recognised as an automated request which triggers Incapsula's challenge page. 

That's what the code you see in the image above - specifically it loads a JavaScript script to request a response.  Basically it's asking you to tick a box to say you are not a bot.

Of course the PQ request can't do this so your query fails.

POSTMAN is a well known piece of software and web firewalls like Incapsula recognise it so it doesn't trigger these challenges.

Unfortunately the same can't be said for Power Query/PBI.

There are only 2 solutions I can think of:

1. Contact the API provider and get them to whitelist your IP address in their Incapsula firewall

2. Contact Incapsula and ask them how you can get around this problem.

Regards

Phil

---

If I answered your question please mark my post as the solution.
If my answer helped solve your problem, give it a kudos by clicking on the Thumbs Up.