Unable to clear lease on resource

Context: 

• Using a Notebook in Fabric to process json files
• Notebook reads json, then writes dataframe to a table, and finally uses mssparkutils to move the processed file
• Fabric capacity recently hit a limit. Many jobs failed.

Our problem:

This fails:

filename = 'Files/problem.json'
destination_dir = 'Files/processed/'

mssparkutils.fs.mv(filename, destination_dir, True)

Here is the error message:

Py4JJavaError: An error occurred while calling z:notebookutils.fs.mv.
: Operation failed: "Precondition Failed", 412, PUT, http://onelake.dfs.fabric.microsoft.com/uuid/uuid/Files/processed?timeout=90, LeaseIdMissing, "There is currently a lease on the resource and no lease ID was specified in the request. RequestId:uuid Time:2024-10-24T15:00:05.1321650Z"
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:231)
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:191)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:464)
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:189)
	at org.apache.hadoop.fs.azurebfs.services.AbfsClient.renamePath(AbfsClient.java:509)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.rename(AzureBlobFileSystemStore.java:986)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.rename(AzureBlobFileSystem.java:456)
	at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl.mvWithinFileSystem(MSFsUtilsImpl.scala:147)
	at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl.$anonfun$mv$2(MSFsUtilsImpl.scala:355)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl.fsTSG(MSFsUtilsImpl.scala:223)
	at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl.$anonfun$mv$1(MSFsUtilsImpl.scala:331)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at com.microsoft.spark.notebook.common.trident.CertifiedTelemetryUtils$.withTelemetry(CertifiedTelemetryUtils.scala:71)
	at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl.mv(MSFsUtilsImpl.scala:331)
	at mssparkutils.IFs.mv(fs.scala:24)
	at mssparkutils.IFs.mv$(fs.scala:20)
	at notebookutils.fs$.mv(utils.scala:12)
	at notebookutils.fs.mv(utils.scala)
	at jdk.internal.reflect.GeneratedMethodAccessor218.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.base/java.lang.Thread.run(Thread.java:829)