Forum Discussion
Web Scraping a table, Identify Changes
Hello everyone!
Im scraping a web table to see when new products are released from a supplier. As they dont have a "new product" list, Im using their "full Product List" table, and want to compare old and new lists to see changes...
I have the PBI pulling the updated table data (latest refresh), but how do you recommend creating the old comparrion table in preparatrion to compare with the new data pull? I tried adding a System Date to the table but I cant make it static and it keeps incrementing with each refresh.
Any ideas?
Power BI is not built to handle prior data and compare with new data. It is an all or nothing refresh.
There are some hacks that can simulate partitioning off data in an ad-hoc incremental refresh, which you can read about here, but I'm not sure I'd put that in production. Also, everytime you republish your PBIX file, it wipes out the history unless you got really fancy and use the ALM Toolkit to preserve partions.
It is a very cool hack. It is a bit too Rube Goldberg for me to rely on for production reporting.
2 Replies
Power BI is not built to handle prior data and compare with new data. It is an all or nothing refresh.
There are some hacks that can simulate partitioning off data in an ad-hoc incremental refresh, which you can read about here, but I'm not sure I'd put that in production. Also, everytime you republish your PBIX file, it wipes out the history unless you got really fancy and use the ALM Toolkit to preserve partions.
It is a very cool hack. It is a bit too Rube Goldberg for me to rely on for production reporting.