Forum Discussion

PRIYAPERURI1's avatar
PRIYAPERURI1
Frequent Visitor
2 years ago

Partial data is pulled even after a successfull refresh.

I'm running code from notebook in fabric to save data to a delta lake. I'm finding issues at different stages of my report's data life cycle.  Flow:  Running code to extract data from API and save ...
Dataflows
v-cboorla-msft's avatar
v-cboorla-msft
Microsoft Employee
2 years ago

Hi PRIYAPERURI1 

 

Thanks for using Microsoft Fabric Community.

 

The issue you are describing seems to be related to data consistency and potential race conditions between your notebook refreshing the data lake and the data mart refreshing itself.

Here are some potential root causes that might be impacting data freshness at various stages.

  • Investigate the script execution. Check logs for errors or warnings during the scheduled run. Verify the API logic. Is there pagination involved? Is there a limit on the number of rows returned at once? Consider implementing retries in your script to handle transient API issues.
  • Three-hour scheduled refresh might be too frequent. Try increasing the notebook refresh interval to minimize potential conflicts with data lake updates.
  • Investigate the data mart refresh process. Is there a limit on the number of rows processed at a time? Check for timeouts or resource limitations during refresh. 

Sometimes below steps might help you.

Temporary Glitch: Clearing cookies and caches can sometimes resolve temporary glitches within the application that might be causing the issue.

Corrupted Data: In rare cases, corrupted data stored in the browser's cache related to Microsoft Fabric might be causing the issue. Clearing the cache removes this potentially problematic data.

Hard Refresh: A hard refresh bypasses the cached version of the webpage and forces the browser to download the latest version from the server. Press Ctrl+Shift+R (Windows) or Cmd+Shift+R (Mac).

Try Microsoft Edge: If you are currently using Chrome or Firefox, try switching to Microsoft Edge to see if the issue persists. Edge is the native browser for Microsoft products and might have better compatibility with Microsoft Fabric.

 

If the issue still persists, please do let us know. Glad to help.

 

I hope this information helps.

 

Thank you.

  • v-cboorla-msft's avatar
    v-cboorla-msft
    Microsoft Employee
    2 years ago

    Hi PRIYAPERURI1 

     

    We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
    In case if you have any resolution please do share that same with the community as it can be helpful to others.
    Otherwise, will respond back with the more details and we will try to help.

     

    Thank you.

    • PRIYAPERURI1's avatar
      PRIYAPERURI1
      Frequent Visitor
      2 years ago

      Hello. I haven't found the solution yet. Still trying to figure out what could be the things that causing the issue. From the provided probable solution, I have checked most of them and following are my responses for them. 

      •  -> I didn’t find any logs for errors or warnings from the refresh history. And API has pagination which is handled by my script successfully when ran manually/schedule my script takes less than a min in most of the cases and sometime more than a min but never more than 5 mins.
      • -> I have changed scheduled refresh to 6 hrs. Not sure if this resolves or not.
      • -> Not sure where to check these limitations for Datamart in fabric.

      Tried these steps as well.Temporary Glitch, Corrupted Data, Hard Refresh,Try Microsoft Edge

      I have cleared the cache and also tried opening the fabric in Microsoft edge but how does any of it helps as the issue persist irrespective of system, website/host and how I’m using it as it is a shared work space.  And point to remember that it happens once in every 2 to 3 weeks, that’s what I have observed so far. I have raised a similar issue earlier, which I missed a meeting and couldn’t connect again regarding this. This was last time when I raised the issue. After that it happened again this time. Case 2405210050004176