Forum Discussion
Old data vs new data comparison
Hey jereaallikko ,
you could do that with incremental refresh. So daily you could just load the latest data and grow your dataset like that. Make sure to add a column with the load date, to be able to handle the snapshots.
The problem is if a load fails or you have to do changes that will reset the dataset you don't have any chance to reload the whole data. For that reason I would try to handle that in the source.
For example, create a stored procedure that is executed daily to copy the data from the source to another table. Like that you can manage the snapshots in the new table. The advantage is, that if you have to reload your data, you have the whole history in the SQL server. Also if you want to use the data in a different way in the future.
If you need any help please let me know.If I answered your question I would be happy if you could mark my post as a solution ✔️ and give it a thumbs up 👍Best regardsDenisBlog: WhatTheFact.biFollow me: twitter.com/DenSelimovic
Hey jereaallikko ,
you could do that with incremental refresh. So daily you could just load the latest data and grow your dataset like that. Make sure to add a column with the load date, to be able to handle the snapshots.
The problem is if a load fails or you have to do changes that will reset the dataset you don't have any chance to reload the whole data. For that reason I would try to handle that in the source.
For example, create a stored procedure that is executed daily to copy the data from the source to another table. Like that you can manage the snapshots in the new table. The advantage is, that if you have to reload your data, you have the whole history in the SQL server. Also if you want to use the data in a different way in the future.
Hi selimovd
I managed to overcome the issue with another internal company database, where all the test results are saved. But this solution could be used if there were no other database.
Thanks for the answer.
Jere
Hey jereaallikko ,
thanks for the reply. If you have a database that has all the data that's the best solution. I'm happy you can use that table 😊
If you think my approach would be an alternative approach I would ask you to mark it as solution. This helps the next person to find a viable solution for a similar problem.
Thank you and best regards
Denis