How to append and store streaming data?

[skipper]

I am developing Twitter analysis for my client. I am able to fetch the data from user timeline.

1. Currently I am having a table that can store 200 records

2. every refresh it fetches the data and 200 records are stored (refreshing the existing)

3. Table will delete the previous data and keep 200 records at any given time

what I need

1. I need another table where it stores historical data

2. Append new data from the refreshed data.

How can I do this?

[V-lianl-msft · Accepted solution]

Hi @skipper ,

You can try incremental refresh.

Table data is filtered by using Power Query date/time parameters with the reserved, case-sensitive names RangeStart and RangeEnd.

When published to the service, with the first refresh operation, the service creates incremental refresh and historical partitions based on incremental refresh policy settings, and then overrides the parameter values to filter and query data for each partition based on date/time values for each row.

This blog may help you:

https://sqldusty.com/2020/05/20/power-bi-incremental-refresh-with-web-api-data-source/ 

Best Regards,
Liang
If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

[selimovd]

Hey @skipper ,

how do you save the data at the moment and where do you store the data?

Best regards

Denis

[skipper]

It is in Power Query at the moment