Forum Discussion
Custom query - Incremental Refresh
Hi Chris.
I finally made it work. Thanks to nexus150 (Ricardo Rincón) and you Chris for all your help.
My issue was that I configured wrong the incremental refresh. I selected an archive data starting longer than rows in my custom table. Let me explain me better:
- I have 9 rows (months intervals) in my custom table but selected 60 months to archive data. So, PollingExpression was looking for months that didn't exist. Changing archive data to 8 rows solved the issue.Also, in my case, the refreshBookmark property was created just after the first reload with applyRefreshPolicy: true, never with false as were the case at your blog.
Finally, I test some different scenarios just to be sure how it work this method and these are my takeaways:
Test #1: using date field with data in just one partition and loading everything from SSMS.
Result: successful, works perfectly.Test #2: Using DetectDataChangesQuery as a scalar, not table.
Result: successful, works perfectly.Test #3: Test with DetectDataChangesQuery as a table loaded in the model.
Result: successful, works perfectly.Test #4: Test without initial reload to applyRefreshPolicy: false.
Result: successful, works perfectly.
Thanks again Chris for your kindness and your patience trying to help me.
Hi Chris, sorry the delay. I was reconstructing the report from zero.
1. When I refresh the custom table I have a query against Snowflake.
2. When I made the full refresh I don't have any query using custom table. It's like PollingExpression isn't recognized by the incremental.
3. When I uploaded the report, I used Tabular Editor 2 to Apply RefreshPolicy in order to create fact table partitions.
4. Next, I configured the PollingExpression to set DetectDataChangesQuery table and saved the model.
5. With partitions created and PollingExpressions configured, then I execute your TMSL script to full refresh. I made this several times reuploading the file, with "Detect Data Changes" selected but also unselected.
6. I checked the table script searching refreshBookmark option but nothing about it. I suppose my fact table isn't recognizing the PollingExpression for some reason.
I even capitalize columns RangeStart and RangeEnd in my DetectDataChangesQuery table in case that capital letters at name was a problem, but doesn't work.
Sorry for all this post, it's just this solution it's crucial to my production model. Thanks for all your time and knowledge.
The only difference I can see between what you're doing and my example is that you are loading the DetectDataChangesQuery query to the dataset and I'm not. Can you disable load for DetectDataChangesQuery and try again?
Hi again Chris.
How do you use DetectDataChangesQuery table without load it into the model? I thought that you did it.
I disable DetectDataChangesQuery load and re try it but keep without generate refreshBookmark property.
Ricardo Rincón it's also trying to help me with this, he pointed that one of my mistakes was configure Archive data starting in Incremental refresh longer than my dates in custom tables. I have 9 rows at custom table and was loading 60 months in incremental.
So, after that I was able to relaunch the full reload with refresh policy true and just in that case I have this.So I have 2 problems still:
- After first full refresh doesn't have refreshBookmarks.
- Incremental load just refresh partition between incremental parameters dates, ignores custom table.No, the engine doesn't need to have the DetectDataChangesQuery query loaded into the dataset in order to use it.
Can you share the M code for your version of DetectDataChangesQuery? I just noticed that from the screenshots you shared, you haven't applied a filter on it using the RangeStart and RangeEnd parameters. Remember that the query should return a table with just one column and one row.
Hi Chris.
I finally made it work. Thanks to nexus150 (Ricardo Rincón) and you Chris for all your help.
My issue was that I configured wrong the incremental refresh. I selected an archive data starting longer than rows in my custom table. Let me explain me better:
- I have 9 rows (months intervals) in my custom table but selected 60 months to archive data. So, PollingExpression was looking for months that didn't exist. Changing archive data to 8 rows solved the issue.Also, in my case, the refreshBookmark property was created just after the first reload with applyRefreshPolicy: true, never with false as were the case at your blog.
Finally, I test some different scenarios just to be sure how it work this method and these are my takeaways:
Test #1: using date field with data in just one partition and loading everything from SSMS.
Result: successful, works perfectly.Test #2: Using DetectDataChangesQuery as a scalar, not table.
Result: successful, works perfectly.Test #3: Test with DetectDataChangesQuery as a table loaded in the model.
Result: successful, works perfectly.Test #4: Test without initial reload to applyRefreshPolicy: false.
Result: successful, works perfectly.
Thanks again Chris for your kindness and your patience trying to help me.