Partial data is pulled even after a successfull refresh.

[PRIYAPERURI1]

I'm running code from notebook in fabric to save data to a delta lake. I'm finding issues at different stages of my report's data life cycle. 

Flow: 

Running code to extract data from API and save that data to lake.

Pulling this data from Lake to a data mart.

And, used this data from Data mart for a BI report. 

So, Now sometimes, I have issue at data lake, only 50 rows are extracted

and other time, same 50 rows are pulled into data mart. Data mart have Scheduled refresh for every 4 hrs and Notebook have Scheduled refresh for every 3 hours . 

Today, I've encountered error at data lake. So, I have manually run the the script and observed that, code is running fine and successfully pulled all of the data, and then I refreshed the data mart by clicking Refresh Now, which also successfully pulled all of the data. 

Last Time I've encountered issue at data mart. At data lake all data is pulled finely, but not at Data mart. So, I have tried manually refreshing the data, which didn't work. So I had to open the data transformations tab to see what's happening. I saw first 99 rows, and then i scrolled down to see all data, when I did that and saved it, Only then I was able to see all of the data. 
same for the first time as well. 

As you can see that here, everytime I need to manually take care of data extraction which is time consumpting and also effecting my report badly. Please help me understand what's happening in here and  resolve it asap!

Thanks,

Priya

[v-cboorla-msft]

Hi @PRIYAPERURI1 

Thanks for using Microsoft Fabric Community.

The issue you are describing seems to be related to data consistency and potential race conditions between your notebook refreshing the data lake and the data mart refreshing itself.

Here are some potential root causes that might be impacting data freshness at various stages.

• Investigate the script execution. Check logs for errors or warnings during the scheduled run. Verify the API logic. Is there pagination involved? Is there a limit on the number of rows returned at once? Consider implementing retries in your script to handle transient API issues.
• Three-hour scheduled refresh might be too frequent. Try increasing the notebook refresh interval to minimize potential conflicts with data lake updates.
• Investigate the data mart refresh process. Is there a limit on the number of rows processed at a time? Check for timeouts or resource limitations during refresh. 

Sometimes below steps might help you.

Temporary Glitch: Clearing cookies and caches can sometimes resolve temporary glitches within the application that might be causing the issue.

Corrupted Data: In rare cases, corrupted data stored in the browser's cache related to Microsoft Fabric might be causing the issue. Clearing the cache removes this potentially problematic data.

Hard Refresh: A hard refresh bypasses the cached version of the webpage and forces the browser to download the latest version from the server. Press Ctrl+Shift+R (Windows) or Cmd+Shift+R (Mac).

Try Microsoft Edge: If you are currently using Chrome or Firefox, try switching to Microsoft Edge to see if the issue persists. Edge is the native browser for Microsoft products and might have better compatibility with Microsoft Fabric.

If the issue still persists, please do let us know. Glad to help.

I hope this information helps.

Thank you.

[v-cboorla-msft]

Hi @PRIYAPERURI1 

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.

Thank you.

[PRIYAPERURI1]

Hello. I haven't found the solution yet. Still trying to figure out what could be the things that causing the issue. From the provided probable solution, I have checked most of them and following are my responses for them. 

•  -> I didn’t find any logs for errors or warnings from the refresh history. And API has pagination which is handled by my script successfully when ran manually/schedule my script takes less than a min in most of the cases and sometime more than a min but never more than 5 mins.
• -> I have changed scheduled refresh to 6 hrs. Not sure if this resolves or not.
• -> Not sure where to check these limitations for Datamart in fabric.

Tried these steps as well.Temporary Glitch, Corrupted Data, Hard Refresh,Try Microsoft Edge

I have cleared the cache and also tried opening the fabric in Microsoft edge but how does any of it helps as the issue persist irrespective of system, website/host and how I’m using it as it is a shared work space.  And point to remember that it happens once in every 2 to 3 weeks, that’s what I have observed so far. I have raised a similar issue earlier, which I missed a meeting and couldn’t connect again regarding this. This was last time when I raised the issue. After that it happened again this time. Case 2405210050004176