Append Data to existing table

[iLikeAzureSQL]

HI everyone, 

I'm loading data from Web Data soruce into a table called LOADTABLE, I need to maintain the hsotorical loads into another table like MYDATA and append data into this table everytime I refresh the data. I looked into PBI incremental option but looks like it's not working with Web Data Source. Do you have other techniques to do this task ?

In PBI Incremental load the source data needs to have datetime to be able to comapre, bu in this case I'm parsting from Web Data.

Thanks,

Pat

[v-jiascu-msft]

Hi Pat,

The incremental refresh has one precondition that is the data should be stored in the source. But it seems the source only has the latest data in your description. 

Did you do to improve the performance of refreshing?

Best Regards,

[iLikeAzureSQL]

I'm not following you! 

I need to connect to Web Data source ( like https://finance.yahoo.com/quote/MSFT) and grab a table and store in a table, the next time I refresh I need to grab new data and append to previously collected records with a timestamp, soemthing like building a qutoa collections over the time.

-Pat

[oscarmqz]

There's no way to do that, I've been struggling with the same... it is so easy to do that in QlikView for example with the incremental load. But power BI works differently, it loads all the data again from the source every time it runs, so the only way to achieve this is that you manually create daily external files to then load them in Power BI. There's a way to create a CSV out of Power BI using an R script, but that will only work if you refresh your dashboard on desktop or personal gateway. R scripts are not supported in the enterprise gateway.

Here's how to use the R script in case it may help you, this will create a new file out of the query with the format History_yymmdd.csv

write.table(dataset, file=paste("C:/Inputs/OTC_Vital_signs/ALL_ERP_HISTORIC/History_",format(Sys.Date(),"%y%m%d"),".csv"), sep=",", row.names=FALSE, append = FALSE, col.names=TRUE)

You can add the R script in the query editor, on the Transform tab; add the step at the very end of your query to generate the  CSV when it is done with the transformations

[v-jiascu-msft]

Hi @iLikeAzureSQL ,

I would suggest you try the Folder connector. You can store the new table in a folder. You will get all the data combined when you refresh the report.

Best Regards,