Partial data is pulled even after a successfull refresh.
I'm running code from notebook in fabric to save data to a delta lake. I'm finding issues at different stages of my report's data life cycle. Flow: Running code to extract data from API and save that data to lake. Pulling this data from Lake to a data mart. And, used this data from Data mart for a BI report. So, Now sometimes, I have issue at data lake, only 50 rows are extracted and other time, same 50 rows are pulled into data mart. Data mart have Scheduled refresh for every 4 hrs and Notebook have Scheduled refresh for every 3 hours . Today, I've encountered error at data lake. So, I have manually run the the script and observed that, code is running fine and successfully pulled all of the data, and then I refreshed the data mart by clicking Refresh Now, which also successfully pulled all of the data. Last Time I've encountered issue at data mart. At data lake all data is pulled finely, but not at Data mart. So, I have tried manually refreshing the data, which didn't work. So I had to open the data transformations tab to see what's happening. I saw first 99 rows, and then i scrolled down to see all data, when I did that and saved it, Only then I was able to see all of the data. same for the first time as well. As you can see that here, everytime I need to manually take care of data extraction which is time consumpting and also effecting my report badly. Please help me understand what's happening in here and resolve it asap! Thanks, Priya