Error using direct query with Spark

[Anonymous]

Hi, I am using direct query (spark connector) to get data from AWS S3 Bucket. I had this error in a visualization:

Error Message: OLE DB or ODBC error: [DataSource.Error] ODBC: ERROR [42000] [Microsoft][Hardy] (80) Syntax or semantic analysis error thrown in server while executing query. Error message from server: org.apache.spark.sql.catalyst.parser.ParseException:
mismatched input '1000001' expecting <EOF>(line 1, pos 11)

== SQL ==
select top 1000001
-----------^^^
`user`,
`manifestation`,
`sentiment`,
`message`,
sum(cast(`favs` as DOUBLE)) as `C1`,
sum(cast(`followers` as DOUBLE)) as `C2`,
sum(cast(`retweets` as DOUBLE)) as `C3`
from
(
select `channel_label`,
`id_date`,
`user`,
`followers`,
`following`,
`manifestation`,
`sentiment`,
`hashtag`,
`message`,
`favs`,
`retweets`
from `twitter`.`fact`
where `hashtag` = ?
) as `ITBL`
group by `user`,
`manifestation`,
`sentiment`,
`message`

The problem occurs because AWS Athena did not accept TOP clause. How can i fix this? Is a possible problem with connector?

[v-deddai1-msft]

Hi @Anonymous ,

Thank you for sharing the solution. Would you please try to accept it as answer to help others find it more quickly.

Best Regards,

Dedmon Dai

[v-deddai1-msft]

Hi @Anonymous ,

Is there any invalid character in the table name? Please refer to https://community.alteryx.com/t5/Alteryx-Designer-Knowledge-Base/Error-running-query-in-Databricks-org-apache-spark-sql-catalyst/ta-p/557451

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Best Regards,

Dedmon Dai

[Anonymous · Accepted solution]

Hi @v-deddai1-msft ,

there isn't an invalid character in the table names. I solved the error using another ODBC connector (Simba Athena) without modifications on schema or table names. This is weird, i think it's an error caused by native Spark connector and i'm curious about this possiblue issue.

Another point is about performance, using Simba Athena the load time in visualizations is very quickly, different from Spark connector which take a time to load visualizations.