Web Scraping a table, Identify Changes

[Anonymous]

Hello everyone!

Im scraping a web table to see when new products are released from a supplier.  As they dont have a "new product" list, Im using their "full Product List" table, and want to compare old and new lists to see changes...

I have the PBI pulling the updated table data (latest refresh), but how do you recommend creating the old comparrion table in preparatrion to compare with the new data pull?  I tried adding a System Date to the table but I cant make it static and it keeps incrementing with each refresh.

Any ideas?

[edhans · Accepted solution]

Power BI is not built to handle prior data and compare with new data. It is an all or nothing refresh.

There are some hacks that can simulate partitioning off data in an ad-hoc incremental refresh, which you can read about here, but I'm not sure I'd put that in production. Also, everytime you republish your PBIX file, it wipes out the history unless you got really fancy and use the ALM Toolkit to preserve partions.

It is a very cool hack. It is a bit too Rube Goldberg for me to rely on for production reporting.

[Anonymous]

Thank you!!!  Sad to see, but it makes sense... thanks for the swift reply.