Forum Discussion
Incremental Refresh
swise001 Hi
Help me understand or correct me if I am wrong .
Incremental refresh main purpose is to reduce the data freshing time by reducing the amount of data to be refreshed (It will not cater my purpose of real time streaming data)
For my purpose of real time streaming data , I need to direct query ( my only concern is performance and datasource stressing )
Have you tried pushing Dataset by Powershell ?
$SqlServer = 'name';
$SqlDatabase = 'datebase';
$SqlConnectionString = 'Data Source={0};Initial Catalog={1};Integrated Security=SSPI' -f $SqlServer, $SqlDatabase;
$SqlQuery = "SELECT * FROM Table;";
$SqlCommand = New-Object -TypeName System.Data.SqlClient.SqlCommand;
$SqlCommand.CommandText = $SqlQuery;
$SqlConnection = New-Object -TypeName System.Data.SqlClient.SqlConnection -ArgumentList $SqlConnectionString;
$SqlCommand.Connection = $SqlConnection;
$SqlConnection.Open();
$SqlDataReader = $SqlCommand.ExecuteReader();
$endpoint = " My end point from the API "
while ($SqlDataReader.Read()) {
$payload = @{ "Data That I get from API info"
}
Invoke-RestMethod -Method Post -Uri "$endpoint" -Body (ConvertTo-Json @($payload))
}
$SqlConnection.Close();
$SqlConnection.Dispose();
The error I get is :
Please comment .
Cheers,
Sangay.
schoden
You are correct with your assessment of incremental refresh. It's mainly a tool to reduce the 'load' on dataset refreshes by only importing a part of the overall dataset.
Unfortunately, once you've created an imported model - there is no way to 'convert' it to a direct query model. You'd have to essentially start over again (and select "direct query") when chosing a data storage mode.
Yes - direct query moves the processing back to the source database - so there are real concerns about performance and datasource stressing - depending on how many users and queries are being applied simultaneously. However, this method will allow you to get 'near-real-time' info - each time a query is run - or a page is refreshed. (In the service this can be 'forced' with a browser refresh).
I do not have much experience with pushing datasets with Powershell. I imagine - if you could programmatically 'force' a new dataset into your model (every few minutes?) - you could also achieve 'near-real-time' results. This would be similar to simply refreshing and re-publishing your PBIX file at a regular cadence. There are no limits that I know of - to how often you can manually re-publish a dataset.
- schoden6 years agoPost Partisan
Regarding the direct query, even though I have set the dataset refresh for 15 mins, the Dashbaord in service doesnt refresh automatically ? the report in the service refreshes every 30 minutes, So like you pointed I put a forced browser refresh( every 1 Minute ) the trick worked but I am seeing limitations as my end user will not have browser refresh installed.
But I am stumbling to understand , even When I set 15 minutes refresh in dataset , no change occurs in dashboard, is it to do with pro license , when its available only in premium?
Thanks for sharing your knowlege with us 😄