Views do not refresh

[KarolinaKa]

Hello!

I'm currently trying to solve the mystery why my scheduled refresh for tables at night do not refresh properly sometimes even though when I run manually that update pipeline, it always works.

Also, sometimes after night scheduled refresh raw tables are updated but regular views (not materialised) made from those raw tables in DWH are not updated.

The only reason I can come is the lack of CPU and memory, but maybe it could be another problem?

[v-hashadapu]

Hi @KarolinaKa , Hope your issue is solved. If it is, please consider marking the answer 'Accept as solution', so others with similar issues may find it easily. If it isn't, please share the details.
Thank you.

[v-hashadapu]

Hi @KarolinaKa , Hope your issue is solved. If it is, please consider marking it 'Accept as solution', so others with similar issues may find it easily. If it isn't, please share the details. Thank you.

[v-hashadapu]

Hi @KarolinaKa , thank you for reaching out to the Microsoft Fabric Community Forum.

Please consider below:

1. As you mentioned, lack of CPU and memory could be a factor, especially if other processes are running concurrently. Reduce the number of queries running simultaneously. This can help manage CPU usage more effectively. Use monitoring tools to keep an eye on CPU and memory usage and adjust your processes accordingly. Run refreshes during times when the system is less busy to ensure more resources are available.
2. Sometimes, network interruptions or timeouts can cause refresh failures.
3. If multiple dataflows are scheduled at the same time, they might be competing for resources. Spreading out the schedules might help.
4. Complex queries can put a heavy load on the system. Simplifying queries or breaking them down into smaller parts might improve performance.
5. Cached data might not always reflect the latest updates. Clearing the cache or forcing a refresh might help.
6. There could be a bug or glitch in the system that's causing intermittent failures. Checking for updates or patches might resolve this.

If this helps, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details.
Thank you.

[KarolinaKa · Accepted solution]

Hey,

it is still a misery to me, because:

a) I turned off other scheduled refreshes and left just this one

b) if I run the pipeline manually, everything works just fine (data lies to DWH)

c) if I run the pipeline with scheduled refresh during the day, everything works just fine

d) if I come in the morning and check nightly scheduled refresh it always shows as 'succeeded' but raw tables in the DWH are not updated

e) My update pipeline architecture: Copy data activity (Mysql to LH data from the last 2 days based on updated_at field) -> Wait activity (180s) -> Upsert BULK script activity (which checks LH and DWH data based on id and retrieves LH new data to the DWH).  Quite simple architecture and works for the other created DWH (quite similar to this one by tables, size structures) while this one lacks to update the tables in the DWH.

f) In the morning I deactivate copy and wait activities in this pipeline and only run manually update script again and everything works just fine (new data lies in the DWH)

g) Crazy, huh?

[v-hashadapu]

Hi @KarolinaKa , thank you for reaching out to the Microsoft Fabric Community Forum.

Sorry to know none of the suggestions helped. Please consider below:

1. Even though you turned off other scheduled refreshes, there might still be other system processes running at night that are consuming resources. Use monitoring tools to keep an eye on CPU, memory, and network usage during the night. Identify any spikes or resource contention that might be affecting the refresh.
2. Scheduled refreshes at night might be more susceptible to network interruptions, especially if there are maintenance activities or lower network performance during those hours. Verify if there are any network maintenance activities or known issues during the night that could be affecting the refresh.
3. There might be a delay in data availability from the source during the night, causing the refresh to pick up incomplete or outdated data. Ensure that the data from the source (MySQL) is available and up-to-date during the night refresh. This can be done by checking the data directly in the source system.
4. If none of the above work, Please consider reaching out to Microsoft Support. You can provide them with all the troubleshooting steps you've already taken, which will help them understand the issue better and provide a resolution. They might be able to identify something specific about your admin account setup or provide a solution that isn't immediately obvious.
   
   Below is the link to create Microsoft Support ticket:
   How to create a Fabric and Power BI Support ticket - Power BI | Microsoft Learn

If this helps, please consider marking it 'Accept as Solution' so others with similar queries may find it more easily. If not, please share the details.
Thank you.