Forum Discussion
Data Flow Gen2 ODBC connection to AWS for Incremental Refresh
Hi, I am in higer Ed and we are not currently on Fabric License.
I have a query against an AWS Athena table using an AWS ODBC Connection.
The table is upward of 17 million rows and approximately 70 columns, many are ID columns. I would like to incrementally update this data.
I have try many ways to do this unsuccessfully for one reason or anther. Either it does not fold or it does not do an incremental refesh.
My current state: dsn, catalog and table are in the source (suggestion in troubleshooting)
Primary query: set to allow staging, it has no destination (Microsoft doc)
Reference query: does not allow staging, it is set to not require folding (it does in Power Query) ,this is where the incremental update is set. I want to refresh an entire year, when the update field has changed.
Destination is a lake house table columns are not dynamic, options set to limit 10
Parameters are RangeStart, RangeEnd points to a query to todays date.
I would really love it if anyone has an issue with AWS and figured this out. (Dates are set as they are per another blog, stating date issues)
This query runs, but it does not appear to do incremental updates. The second time I run it after adding additional data is longer then the first.
// 1. Format dates for Athena
StartText = DateTime.ToText(RangeStart, "yyyy-MM-dd"),
EndText = DateTime.ToText(RangeEnd, "yyyy-MM-dd"),
// 2. The SQL Statement with manual folding built-in
FullQuery = "
SELECT
journal_line_wid,
as_entered_company, (lots of fields)
FROM edl_workday_fin.cur_journal_transactions_vw
WHERE accounting_date >= (select date '" & StartText & "') and accounting_date < (select date '" & EndText & "')",
// 3. The Actual Data Connection
Source = Odbc.Query("dsn=DSNName;Catalog=AwsDataCatalog;Schema=SchemaName;",
FullQuery
)
in
Source
your incremental refresh is not working because Dataflow Gen2 incremental refresh requires query folding at the source, and ODBC to AWS Athena does not properly support folding in Fabric.
When you use Odbc.Query() with a manually constructed SQL string, Fabric treats it as a native query. Native queries break folding. Even if Power Query preview looks fine, the Fabric service cannot push partition filters back during refresh. So every refresh scans everything again. That is why your 2nd run is slower. It is reading data again, not refreshing partitions.
Key issues in your setup:
-
Odbc.Query() = no folding recognition by Fabric incremental engine. -
“Does not require folding” on reference query = incremental refresh will not partition.
-
Manual SQL string with RangeStart/RangeEnd = Fabric cannot inject partition filters dynamically.
-
Athena ODBC driver does not expose metadata in a way Fabric can fold date filters automatically.
Incremental refresh in Dataflow Gen2 works only when:
-
The source supports folding.
-
The date filter step using RangeStart and Range End is directly applied to a foldable source step.
-
You do NOT use a hard-coded SQL statement.
With Athena via ODBC, this rarely works.
As a fix, I would suggest move ingestion using:
-
AWS glue job -> parquet partitioned by date
-
or replicate using ADF/fabric pipelines
-
Then use fabric Lakehouse incr refresh on Parquet/Delta (fully supported).
Right now you are trying to force incremental refresh on a connector that does not properly support folding. That is the root problem.
-
3 Replies
- Vinodh247
Super User
your incremental refresh is not working because Dataflow Gen2 incremental refresh requires query folding at the source, and ODBC to AWS Athena does not properly support folding in Fabric.
When you use Odbc.Query() with a manually constructed SQL string, Fabric treats it as a native query. Native queries break folding. Even if Power Query preview looks fine, the Fabric service cannot push partition filters back during refresh. So every refresh scans everything again. That is why your 2nd run is slower. It is reading data again, not refreshing partitions.
Key issues in your setup:
-
Odbc.Query() = no folding recognition by Fabric incremental engine. -
“Does not require folding” on reference query = incremental refresh will not partition.
-
Manual SQL string with RangeStart/RangeEnd = Fabric cannot inject partition filters dynamically.
-
Athena ODBC driver does not expose metadata in a way Fabric can fold date filters automatically.
Incremental refresh in Dataflow Gen2 works only when:
-
The source supports folding.
-
The date filter step using RangeStart and Range End is directly applied to a foldable source step.
-
You do NOT use a hard-coded SQL statement.
With Athena via ODBC, this rarely works.
As a fix, I would suggest move ingestion using:
-
AWS glue job -> parquet partitioned by date
-
or replicate using ADF/fabric pipelines
-
Then use fabric Lakehouse incr refresh on Parquet/Delta (fully supported).
Right now you are trying to force incremental refresh on a connector that does not properly support folding. That is the root problem.
-
- nielsvdc
Super User
There is an Amazon Athena connector available or Dataflow Gen2. Have you tried that connector instead of the generic ODBC pattern?
- v-prasare
Community Support
Hi Suzanne_Shuford,
We would like to confirm if our community members answer resolves your query or if you need further help. If you still have any questions or need more support, please feel free to let us know. We are happy to help you.
Thank you for your patience and look forward to hearing from you.
Best Regards,
Prashanth Are
MS Fabric community support