The ultimate Fabric, Power BI, SQL, and AI community-led learning event. Save €200 with code FABCOMM.
Get registeredEnhance your career with this limited time 50% discount on Fabric and Power BI exams. Ends August 31st. Request your voucher.
I have implemented the incremental refresh for 3 tables in my data model. But one table is loading the data partially instead of loading all the rows. Below are the details:
Source: Databricks
Load Mode: Import
Refresh Frequency: Once every day
The data is quite huge (4 Million rows/day) and it's not possible to load 1 year's of data onto a desktop and publish it every day as it was attempting full refresh and it would time out. To tackle this issue we published the data model with limited rows (Using StartRange and StartEnd filter) and then updated the historical partitions through the SSMS XMLA endpoint.
The below image shows my partitions (Notice the row count for 2023Q10213):
I have scheduled it to refresh once every day. The problem is that when a refresh happens all tables except one get refreshed properly but for one table the rows are loaded partially. When I delete the partition, recreate it, and load it manually then it loads all the rows in the partition. You can see below image, compared to the above image, that rows are now fully loaded after I deleted, recreated, and loaded the partition manually (2023Q10213):
Can someone please guide me on how to debug this issue as well as what's the reason behind this behavior? I am attaching my increment policy screenshot below:
Thank you!
User | Count |
---|---|
43 | |
16 | |
12 | |
11 | |
9 |
User | Count |
---|---|
50 | |
31 | |
22 | |
17 | |
15 |