Semantic model refresh missing dataflow data

[agusmba]

Following the good practices, I have set up an extraction dataflow layer, loading data from two different sources (two different extraction dataflows). On the transformation layer, a third dataflow is merging a couple of tables from the first two. Then the semantic model is loading the table from the third dataflow. Note that this is a small 12 row table!

The first time I deployed the new table, all went fine. However, after I modified data on one of the original sources (Excel), the new data is not appearing after multiple dataflow refreshes and semantic model refreshes. I have been trying for the last three days: unlinking, relinking, creating new modifications... to no avail. The latest data changes to that table simply won't appear. I don't want to delete completely the report since it is part of further dashboards, goals, etc.

Note that if I explore the dataflow online or if I refresh the report with Power BI Desktop, the new data seems to be there, it's just that the semantic model online refresh is not picking them up. However data changes on other bigger tables refresh without problems.

EDIT: removed references to pipelines, since this problem is happening in the dev workspace also.

[agusmba]

This is happenning in most of my online semantic models getting data from dataflows. Not all data updates are being refreshed in the semantic model (the changes are there if I edit the dataflow).

[v-xiaoyan-msft]

Hi @agusmba ,

We noticed that you mentioned that the problem occurred after you modified the data in the Excel data source.
Have you tried republishing the report from Power BI Desktop after updating the data?

Best regards.
Community Support Team_Caitlyn

[agusmba]

Hi @v-xiaoyan-msft ,

Yes I've tried several things to see if I can get the online semantic model to update with the latest dataflow data, but nothing seems to work.

I've tried republishing the report, making changes to the report, both visuals and to the model (like ordering the table with missing data in power query for instance), but the online semantic model just won't pick up the new data (modified rows in a small table).

If I refresh my Power BI Desktop report, against the same dataflows, I can see the new data.

Of course If I save the local report and republish it, the new data will also be online, but that defeats the purpose of refreshing the model online and renders pipelines unusable.

Is there a way to "force" a full refresh of the online model?

Thank you and best regards,

   Agustín.

[agusmba]

To further investigate this issue, I downloaded one of the problematic reports (pbix) that gets its data from a dataflow in the same workspace (PROD).

I opened the downloaded report in Power BI Desktop and saw that it had pending model changes (there were no pending changes when I published it to DEV). If I accepted them, I lost the whole model. If I rejected it, all seemed fine.

However even refreshing from Desktop, the new data was not appearing (even refreshing from one of the tabs where the missing data was supposed to appear). If I edited the query, the preview showed the correct data (from the dataflow).

The only way to get the new data into the local report was to select the specific table and do a "refresh data" for that specific table (a general "refresh all" did not work).

Again this workaround is unusable in any production environment where I publish several reports and use pipelines to deploy to TEST and PROD environments, unless the recommendation is to ignore all the pipelines deployments and manually publish from local to PROD, and do all refreshes locally (table by table...).

While someone at Microsoft finds and fixes the normal on-demand and scheduled refreshes, how can we "force" a full refresh of an online semantic model that queries dataflows? (the model is not incremental, nor does it have cache configured).

BTW, when I said I encountered this issue recently, this is actually the cause of an unexplicable problem with summarizing data incorrectly that I've been stumped with since late november/december. The thing is that now I'm experiencing the same problem in more reports as I've been relying more and more on dataflows (maybe switching to them was the wrong move).

[v-xiaoyan-msft]

Hi @agusmba ,

For the time being, no other users have reported similar experiences to yours. As we are not allowed to collect user logs, it is difficult to investigate further based on the information available. In order to better solve your problem.Based on the above information, if you are a Power BI Pro licensee, you can create a support ticket for free and a dedicated Microsoft engineer will come to solve the problem for you.
It would be great if you continue to share in this issue to help others with similar problems after you know the root cause or solution.

The link of Power BI Support: Support | Microsoft Power BI

For how to create a support ticket, please refer to How to create a support ticket in Power BI - Microsoft Power BI Community

Best Regards,
Community Support Team _ Caitlyn

[agusmba]

In the end, after much head scratching, it was a simple misconfiguration problem on a couple of tables out of a hundred.

In order to avoid reviewing manually a large number of tables to check if any has been excluded from the report refresh, you can save your report as a project (pbip) and search in the code for the attribute excludeFromModelRefresh. If present (and true), it will point to tables that will suffer from this issue.

It's actually not an issue since the platform is only doing as instructed 🤦.

Best regards,

  A.M.