Reading a OneLake Shortcut - Getting frequent errors

This happens with different tables and does not happen every single time we try to load the Delta Table in our Notebook, but it will happen 7 out of 10 times. 

For clarity on the Looping part. We just have a list of table names (all strings) and we loop over them going through a read operation, filter and then save to default Lakehouse. While I am looking at the option of just having a parametrized Notebook, I have the theory that it will not solve the issue for the shortcut.

Here is an error I got today:

Py4JJavaError: An error occurred while calling o8318.count. 
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 in stage 1544.0 failed 4 times, most recent failure: Lost task 8.3 in stage 1544.0 (TID 15529) (vm-d5775677 executor 2): org.apache.spark.SparkFileNotFoundException: Operation failed: "Not Found", 404, HEAD, http://onelake.dfs.fabric.microsoft.com/<workspace>/<lakehouse>.Lakehouse/Tables/<shortcut_table>/part-00000-45f6a097-1230-45cc-8dc7-dc1a1e2dafeb-c000.snappy.parquet?upn=false&action=getStatus&timeout=90 

It is possible the underlying files have been updated. You can explicitly invalidate the cache in Spark by running 'REFRESH TABLE tableName' command in SQL or by recreating the Dataset/DataFrame involved.