Forum Discussion
Push Data Set - Data Looks Wierd
Ideally your data source has things like LIMIT and OFFSET that you can use to paginate. If not then do the pagination in Powershell.
I wouldn't worry about the record identifier, it should work without that. What you seem to be missing is a UTC timestamp field. That is mandatory for push datasets.
Here is an example using Powershell (I use that to monitor gateway cluster member health)
#capture stats
$mem = (Get-Counter '\Memory\Available MBytes').CounterSamples.CookedValue
$proc = (Get-Counter '\Processor(_Total)\% Processor Time').CounterSamples.CookedValue
$disk = (Get-Counter '\LogicalDisk(C:)\% free space').CounterSamples.CookedValue
$date = (Get-Date).ToUniversalTime().ToString('yyyy-MM-ddTHH:mm:00.000Z')
$payload = @{
"Host" = $env:computername
"Timestamp" = $date
"Available Memory in MB" = $mem
"Processor Load %" = $proc
"% Free on C:" = $disk
}
#enforce TLS1.2
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
#push URL
$endpoint = "https://api.powerbi.com/beta/<tenant id>/datasets/<dataset id>/rows?key=<push key>"
Invoke-RestMethod -Method Post -Uri "$endpoint" -Body (ConvertTo-Json @($payload))
I'm not entirely sure a UTC is required in this case. I've got the following code working well with a single entry and data now seems to be coming in the way it should:
$headers = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers.Add("Authorization", "Your Authorization")
$response = Invoke-RestMethod 'Where the data is' -Method 'GET' -Headers $headers
Write-Output ""
Write-Output ""
$headers2 = New-Object "System.Collections.Generic.Dictionary[[String],[String]]"
$headers2.Add("Content-Type", "application/json")
write-output $body
$response2 = Invoke-RestMethod 'Where the data needs to go' -Method 'POST' -Headers $headers2 -Body (ConvertTo-Json @($response.value))
$response2 | ConvertTo-Json
The issue now is the dataset that I'm going to be pushing is roughly 24k lines worth. Power bi has a limitation of 10k lines per post and only 5 queued posts at a time. What do you think is the best way to divvy up my response from the Invoke get method would be?