Forum Discussion
How to append and store streaming data?
I am developing Twitter analysis for my client. I am able to fetch the data from user timeline.
1. Currently I am having a table that can store 200 records
2. every refresh it fetches the data and 200 records are stored (refreshing the existing)
3. Table will delete the previous data and keep 200 records at any given time
what I need
1. I need another table where it stores historical data
2. Append new data from the refreshed data.
How can I do this?
Hi skipper ,
You can try incremental refresh.
Table data is filtered by using Power Query date/time parameters with the reserved, case-sensitive names RangeStart and RangeEnd.
When published to the service, with the first refresh operation, the service creates incremental refresh and historical partitions based on incremental refresh policy settings, and then overrides the parameter values to filter and query data for each partition based on date/time values for each row.
This blog may help you:
https://sqldusty.com/2020/05/20/power-bi-incremental-refresh-with-web-api-data-source/
Best Regards,
Liang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
3 Replies
It is in Power Query at the moment
Hi skipper ,
You can try incremental refresh.
Table data is filtered by using Power Query date/time parameters with the reserved, case-sensitive names RangeStart and RangeEnd.
When published to the service, with the first refresh operation, the service creates incremental refresh and historical partitions based on incremental refresh policy settings, and then overrides the parameter values to filter and query data for each partition based on date/time values for each row.
This blog may help you:
https://sqldusty.com/2020/05/20/power-bi-incremental-refresh-with-web-api-data-source/
Best Regards,
Liang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.
