Forum Discussion
Google Analytics connector missing startdate/enddate parameters
I thought this had been asked before, but I didn't find a thread via search.
A GA connection has a query pattern like so:
let
Source = GoogleAnalytics.Accounts(),
account = Source{[Id="654321"]}[Data],
webprop = account{[Id="UA-654321-1"]}[Data],
profile = webprop{[Id="123456"]}[Data],
gadata = Cube.Transform(
profile,
{
{
Cube.AddAndExpandDimensionColumn,
"ga:date",
{"ga:date"},
{"Date"}
},
{
Cube.AddMeasureColumn,
"Sessions",
"ga:sessions"
}
}
),
in
gadataThat seems to provide coverage for the GA API except for the start date and end date parameters. Are those parameters exposed in some way? If not, how does one specify a query like "users last month". Would it be possible to add them as optional parameters to the GoogleAnalytics.Accounts() or Cube.Transform() function?
39 Replies
- artemmuntianuFrequent Visitor
Our solution to un-sampling is to break a month/year query to many daily queries, and aggregate the data later on the client-side.
Here is a code
let Source = GoogleAnalytics.Accounts(), #"1" = Source{[Id="999"]}[Data], #"2" = #"1"{[Id="999"]}[Data], #"3" = #"2"{[Id="999"]}[Data], #"newCube" = Cube.Transform(#"3", { {Cube.AddAndExpandDimensionColumn, "ga:date", {"ga:date"}, {"Date"}}, {Cube.AddAndExpandDimensionColumn, "ga:deviceCategory", {"ga:deviceCategory"}, {"Device Category"}}, {Cube.AddAndExpandDimensionColumn, "ga:userAgeBracket", {"ga:userAgeBracket"}, {"Age"}}, {Cube.AddMeasureColumn, "Sessions", "ga:sessions"} }), combinedData = Table.Combine({ Table.SelectRows(#"newCube", each ([Date] = #date(2017, 1, 1))), Table.SelectRows(#"newCube", each ([Date] = #date(2017, 1, 2))), ... {days in between} ... Table.SelectRows(#"newCube", each ([Date] = #date(2018, 3, 4))), Table.SelectRows(#"newCube", each ([Date] = #date(2018, 3, 5))) }) in combinedDataWhen we run that code it makes PowerBI to send 429 GA requests. One request per day.
So, Power BI is actually requests GA with proper startdate and enddate params.
One problem still remains - when you combine data on the client side it becomes a bit inaccurate in terms of unique counters.
- pqianMicrosoft Employee
PowerQuery models this with filters:
#"Filtered Rows" = Table.SelectRows(gadata, each Date.IsInPreviousWeek([Date]))
- intrasightHelper IV
Please correct me if I'm wrong, but I don't think this can be done with filter semantics.
Let me give a more realistic example
let Source = GoogleAnalytics.Accounts(), account = Source{[Id="654321"]}[Data], webprop = account{[Id="UA-654321-1"]}[Data], profile = webprop{[Id="123456"]}[Data], gadata = Cube.Transform( profile, { { Cube.AddMeasureColumn, "Users", "ga:users" } } ), in gadataAs you see, there is no date dimension in the query. Because I want the value of the metrics over a date range. I need metrics (such as ga:users) which cannot be aggregated client-side. So I need the connector to support the missing start and end date parameters.
- pqianMicrosoft Employee
I'm not sure what you mean by Cannot aggregate client-side. If the date filter is in the expected range, we will fold that into the start-date and end-date parameter of the request URL. For example, this query:
#"Added Items" = Cube.Transform(cube, {{Cube.AddAndExpandDimensionColumn, "ga:date", {"ga:date"}, {"Date"}}, {Cube.AddMeasureColumn, "Sessions", "ga:sessions"}, {Cube.AddMeasureColumn, "Users", "ga:users"}}), #"Filtered Rows" = Table.SelectRows(#"Added Items", each [Date] >= #date(2016, 1, 27) and [Date] <= #date(2016, 2, 4))will be folded into
?...&metrics=ga:sessions,ga:users&start-date=2016-01-27&end-date=2016-02-04&start-index=1
If you don't want to see the date dimension, you can collapse it later. The filter stays active.
= Cube.CollapseAndRemoveColumns(#"Filtered Rows", {"Date"})
- intrasightHelper IV
Hi,
For some reason the discussion forum won't let me respond to your most recent post, so I'm responding to this one.
By client-side, I mean that the sum is being calculated by the client - Power BI in this case - as opposed to server-side (in GA). Query folding to GA cannot work if the GA query includes the date dimension, which it still does after adding the filter. You'll see this if you look at the GA API call being made.
The correct answer from GA for my query "users last month" is 893. Power BI is reporting 1034, which is the sum of the users on each of the 31 days. "users" and perhaps half of the other metrics cannot be aggregated client-side, and so Power BI is reporting incorrect values. For query folding to work against GA, it would have to remove the date dimension from the API call - which is clearly not happening.- intrasightHelper IV
Here's the specifics. Here's the query:
let Source = GoogleAnalytics.Accounts(), GaAccount = Source{[Id="xxxx"]}[Data], GaWebProp = GaAccount{[Id="UA-xxxx-1"]}[Data], Profile = GaWebProp{[Id="13844713"]}[Data], #"Added Items" = Cube.Transform(Profile, {{Cube.AddAndExpandDimensionColumn, "ga:date", {"ga:date"}, {"Date"}}, {Cube.AddMeasureColumn, "Users", "ga:users"}}), #"Filtered Rows" = Table.SelectRows(#"Added Items", each [Date] >= #date(2016, 1, 1) and [Date] <= #date(2016, 1, 31)), #"Collapsed and Removed Columns" = Cube.CollapseAndRemoveColumns(#"Filtered Rows", {"Date"}) in #"Collapsed and Removed Columns"And here's the three GA API queries made:
https://www.googleapis.com/analytics/v3/data/ga?ids=ga:13844713&filters=ga:date==20160101,ga:date==20160102,ga:date==20160103,ga:date==20160104,ga:date==20160105,ga:date==20160106,ga:date==20160107,ga:date==20160108,ga:date==20160109,ga:date==20160110,ga:date==20160111,ga:date==20160112,ga:date==20160113,ga:date==20160114,ga:date==20160115,ga:date==20160116,ga:date==20160117,ga:date==20160118,ga:date==20160119,ga:date==20160120,ga:date==20160121,ga:date==20160122,ga:date==20160123,ga:date==20160124,ga:date==20160125,ga:date==20160126,ga:date==20160127,ga:date==20160128,ga:date==20160129,ga:date==20160130,ga:date==20160131&metrics=ga:users&start-date=2016-01-01&end-date=2016-01-31&start-index=1&max-results=1000 https://www.googleapis.com/analytics/v3/data/ga?ids=ga:13844713&filters=ga:date==20160101,ga:date==20160102,ga:date==20160103,ga:date==20160104,ga:date==20160105,ga:date==20160106,ga:date==20160107,ga:date==20160108,ga:date==20160109,ga:date==20160110,ga:date==20160111,ga:date==20160112,ga:date==20160113,ga:date==20160114,ga:date==20160115,ga:date==20160116,ga:date==20160117,ga:date==20160118,ga:date==20160119,ga:date==20160120,ga:date==20160121,ga:date==20160122,ga:date==20160123,ga:date==20160124,ga:date==20160125,ga:date==20160126,ga:date==20160127,ga:date==20160128,ga:date==20160129,ga:date==20160130,ga:date==20160131&metrics=ga:users&dimensions=ga:date&start-date=2016-01-01&end-date=2016-01-31&start-index=1&max-results=1000 https://www.googleapis.com/analytics/v3/data/ga?ids=ga:13844713&metrics=ga:users&dimensions=ga:date&start-date=2009-01-01&end-date=2016-02-04&start-index=1&max-results=1000
The first filters in all the dates (which is redundant due to start-date and end-date, and has no date dimension. This query if run in GA returns users=893
The second again lists all dates but also has date dimension. This query if run in GA returns users=1034
The third doesn't list dates but has current date as end-date. I don't know why it runs this query - I didn't check GA directly.
In Power BI, the value returned is users=1034. The correct value is users=893.
- pqianMicrosoft Employee
intrasight, interesting. I'm not very familier with GA's API, I can talk to the dev who worked on this. Particularly, I'm a bit surprised that the inclusion of Date dimension breaks the aggregation on GA.
PowerQuery's multidimensional model had limited support for additional parameters to the cube. We've only began adding it for SAP and AS. So for the GA connector, the start/end date parameter is automatically populated via the filter on date dimension. I think there's could be a bug in the PowerQuery folding logic, when the date dimension is collapsed, we should not have sent &dimension=ga:date.
I'll follow up on this and get back to you.
- intrasightHelper IV
It's the same reason that it would be wrong in MDX - you can't aggregate client-side. Have someone who knows MDX look at the GA integration and I'm sure they can straighten it out. Or have them contact me.
I did see the additional parameters in SAP and AS, and thought "that's the solution".
Thanks for investigating.
- pqianMicrosoft Employee
Thanks for reporting the issue. We are now tracking this issue and will address it soon
- artemmuntianuFrequent Visitor
Our solution to un-sampling is to break a month/year query to many daily queries, and aggregate the data later on the client-side.
Here is a code
let Source = GoogleAnalytics.Accounts(), #"1" = Source{[Id="999"]}[Data], #"2" = #"1"{[Id="999"]}[Data], #"3" = #"2"{[Id="999"]}[Data], #"newCube" = Cube.Transform(#"3", { {Cube.AddAndExpandDimensionColumn, "ga:date", {"ga:date"}, {"Date"}}, {Cube.AddAndExpandDimensionColumn, "ga:deviceCategory", {"ga:deviceCategory"}, {"Device Category"}}, {Cube.AddAndExpandDimensionColumn, "ga:userAgeBracket", {"ga:userAgeBracket"}, {"Age"}}, {Cube.AddMeasureColumn, "Sessions", "ga:sessions"} }), combinedData = Table.Combine({ Table.SelectRows(#"newCube", each ([Date] = #date(2017, 1, 1))), Table.SelectRows(#"newCube", each ([Date] = #date(2017, 1, 2))), ... {days in between} ... Table.SelectRows(#"newCube", each ([Date] = #date(2018, 3, 4))), Table.SelectRows(#"newCube", each ([Date] = #date(2018, 3, 5))) }) in combinedDataWhen we run that code it makes PowerBI to send 429 GA requests. One request per day.
So, Power BI is actually requests GA with proper startdate and enddate params.
One problem still remains - when you combine data on the client side it becomes a bit inaccurate in terms of unique counters.
- AnonymousNot applicable
Thank you for the above code. This works to circumvent the sampling issue!
I use a code like this:
let Source = GoogleAnalytics.Accounts(), #"1" = Source{[Id="999"]}[Data], #"2" = #"1"{[Id="999"]}[Data], #"3" = #"2"{[Id="999"]}[Data], #"Added Items" = Cube.Transform(#"3", { {Cube.AddAndExpandDimensionColumn, "ga:channelGrouping", {"ga:channelGrouping"}, {"Default Channel Grouping"}}, {Cube.AddAndExpandDimensionColumn, "ga:customfield", {"ga:customfield"}, {"Customfield"}}, {Cube.AddAndExpandDimensionColumn, "ga:yearMonth", {"ga:yearMonth"}, {"Month of Year"}}, {Cube.AddMeasureColumn, "Revenue", "ga:transactionRevenue"}, {Cube.AddMeasureColumn, "Transactions", "ga:transactions"} }), #"combinedData" = Table.Combine({ Table.SelectRows(#"Added Items", each ([Month of Year] = "201803")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201804")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201805")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201806")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201807")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201808")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201809")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201810")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201811")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201812")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201801")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201802")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201903")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201904")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201905")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201906")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201907")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201908")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201909")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201910")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201911")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201912")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201901")), Table.SelectRows(#"Added Items", each ([Month of Year] = "201902")), Table.SelectRows(#"Added Items", each Text.Contains([Month of Year], "2017")) }) in combinedDataThis works, but for performance reasons and scalability I'm looking for an improvement over this as I now query again and again also the historical data. Ideally I would like to create a combined datasource that contains all data up till and including last month. And then have a seperate connection for the data for this month. And combine the two. That way I don't have to send all seperate monthly requests for the historic data anymore, which saves on GA requests. That historic datasource should of course update as soon as a new month starts, so for instance when August starts my historic dataset should automatically contain data up till and inclusing July 2018 (as I aim to avoid manual tasks).
Is this possible in the query editor? Anybody any ideas?
- AnonymousNot applicable
Hi all,
I just wanted to see if there has been any updates on this topic or if anyone found a way around this?
I am getting the correct figures by applying the above mentioned steps for date and using collapse however I have to filter my data which contains a model name in all the page paths. This works really well on the GA query builder but when I add the "contains" filter in Power BI I don't get the correct visitors numbers. It would allow me to collapse and remove as it says that it is not a cube.