Forum Discussion
ODBC and Large Data Sets
>>It is like working with tinker toys compared to Power Query.
It's funny you say that. As I mentioned I'm a big proponent of Power BI. Unfortunately for me, the tinker toy tool (Tableau) is actaully returning data. Because of the way Power BI writes queries, it's just blowing up. I don't need to look at the entire picture. One tool (Power BI) just flat doesn't work for large data sets via ODBC without hand feeding it data via custom SQL. Even this is pretty hard because if I use a limit clause, Power BI then proceeds to wrap this and aggregate only on my limited number of records. I can then fully develop my report, but I have to remove the limit clause at some point at which time Power BI proceeds to try to download 300M records again thereby choking. Final victory - "tinker toy" Tableau...
Thanks again for your time. I was already pretty sure that there was not a good solution I was just looking for confirmation.
Edit: grammar.
Here is a bit of background as to what happens when I actually try to use Power BI. I also have a discussion going on the Power Query board. They don't seem to have an answer either. The example below explains what happens when I try to filter and group by in sequence. I've included the SQL generated as well as the M code. This is mainly for future reference. I don't expect anyone to actually diagnose this. At some point though I'd love to talk to the query engine development team...
It is sending the following query when coming into the Power Query editor in preview mode (10 min wait):
SELECT foo1, foo2,...foo74 FROM bar
Note: NO LIMIT CLAUSE! Tableau uses a limit clause and returns in 5 seconds.
Then I ask for only my two attributes and two measures (2-3 minute wait):
SELECT foo1, foo2, foo3, foo4 FROM bar
Again: no limit clause
Then I filter where one measure is not null (another 2-3 minute wait):
SELECT foo1, foo2, foo3, foo4 FROM bar WHERE foo3 IS NOT NULL
Then I do a group by both attributes with SUM on both measures. I can't provide this query from Power BI as it blows up my client before finishing. It's trying to return all 300M records to my client so it's pretty obvious that it did about the same thing as immediately above except without whatever limits it normally uses.
Tableau sends the following, which returns in about 5 seconds:
SELECT foo1, foo2, SUM(foo3) foo3, SUM(foo4) foo4 FROM bar WHERE FOO3 IS NOT NULL GROUP BY foo1, foo2 LIMIT 1000
Here is the M code from Power BI:
let
Source = Odbc.DataSource("dsn=datalake-poc-athena-64", [HierarchicalNavigation=true]),
AwsDataCatalog_Database = Source{[Name="AwsDataCatalog",Kind="Database"]}[Data],
hmda_Schema = AwsDataCatalog_Database{[Name="hmda",Kind="Schema"]}[Data],
hmda_lar_Table = hmda_Schema{[Name="hmda_lar",Kind="Table"]}[Data],
#"Removed Other Columns" = Table.SelectColumns(hmda_lar_Table,{"action_taken_name", "agency_name", "applicant_income_000s", "loan_amount_000s"}),
#"Filtered Rows" = Table.SelectRows(#"Removed Other Columns", each ([applicant_income_000s] <> null)),
#"Grouped Rows" = Table.Group(#"Filtered Rows", {"action_taken_name", "agency_name"}, {{"Income", each List.Sum([applicant_income_000s]), type number}, {"Loan", each List.Sum([loan_amount_000s]), type number}})
in
#"Grouped Rows"