Model with 2.8 Million Rows Not Auto-Refreshing, Truncated

[Moehimby]

Hey all, appreciate your help!

I created a simple semantic model, a connection to a SQL table that is set up in our gateway, it's basically just a complete drop of all the contents in the table which accounts to 2.8 million records. The model is the data source of 2 paginated reports I've built out and works well, it's super quick to pull the reports compared to what we used to get with older solutions.

If I refresh the model in desktop, no problem. If I manually refresh the model in service, no problem.

Only when I set up an AUTO REFRESH to run in the early morning is there a problem, it doesn't seem to pull all the records down and users complain of missing data until I go back in myself and manually re-run a refresh.

There are not errors reported ever. I've scheduled a auto-refresh outside of times when I have other refreshes happening to avoid timeouts (although I understand the timeout in Premium is like 5 hours, so I can't see it being that).

Anyone got any ideas? It's a drag manually refreshing this every day and I do want to take a vacation at some time.

[v-lgarikapat · Accepted solution]

Hi @Moehimby ,

Thanks for reaching out to the Microsoft fabric community forum.

@lbendlin 

Thanks for your prompt response

@Moehimby 

To troubleshoot your Power BI model not fully refreshing during scheduled auto refresh, start by checking the refresh history in the Power BI Service compare the row count, duration, and dataset size between manual and scheduled runs to spot inconsistencies. Ensure there's no time-based logic like DateTime.LocalNow() in Power Query that could behave differently overnight, and double check that you're not unintentionally using incremental refresh or filters. Look into your on-prem gateway's CPU/memory usage and logs during the scheduled window in case it's silently throttling or timing out and consider restarting the gateway if it’s been running long. On the SQL side, check for long running ETL jobs or table loads that might still be in progress when the Power BI refresh kicks off, potentially leading to partial data loads if that’s the case, reschedule the refresh for later or use a view that only returns completed data. Also, verify that your SQL server isn’t terminating idle or long running queries during unattended hours. As a workaround, try wrapping your SQL logic in a stored procedure for better control, or simulate an early morning refresh in Power BI Desktop via Task Scheduler to see if the issue is specific to the service.

We truly appreciate your continued engagement and thank you for being an active and valued member of the community.

If you're still experiencing any challenges, please don’t hesitate to reach out  we’d be more than happy to assist you further.

We look forward to hearing from you.

Best regards,
Lakshmi

[Moehimby]

> check for long running ETL jobs or table loads that might still be in progress when the Power BI refresh kicks off, potentially leading to partial data loads if that’s the case, reschedule the refresh for later or use a view that only returns completed data.

This is what was happening, I was refreshing in the middle of the source sql table refresh, I pushed the auto refresh time out to a later time and it worked! Thanks for the reply!

[v-lgarikapat]

Hi @Moehimby ,

Glad to hear that worked! Refreshing during an ongoing ETL can definitely cause partial data issues.

Best Regards,

Lakshmi

[lbendlin]

 it's basically just a complete drop of all the contents in the table 

or maybe there are some datetime filters?  Remember that the Power BI service runs on UTC, so "Today"  means different things.

[Moehimby]

Understood but that's not it. It's a significant omittence of a lot of data from the past, data spans 30+ years.