Unable to clear lease on resource

[cailen_cg]

Context: 

• Using a Notebook in Fabric to process json files
• Notebook reads json, then writes dataframe to a table, and finally uses mssparkutils to move the processed file
• Fabric capacity recently hit a limit. Many jobs failed.

Our problem:

This fails:

filename = 'Files/problem.json'
destination_dir = 'Files/processed/'

mssparkutils.fs.mv(filename, destination_dir, True)

Here is the error message:

Py4JJavaError: An error occurred while calling z:notebookutils.fs.mv.
: Operation failed: "Precondition Failed", 412, PUT, http://onelake.dfs.fabric.microsoft.com/uuid/uuid/Files/processed?timeout=90, LeaseIdMissing, "There is currently a lease on the resource and no lease ID was specified in the request. RequestId:uuid Time:2024-10-24T15:00:05.1321650Z"
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.completeExecute(AbfsRestOperation.java:231)
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.lambda$execute$0(AbfsRestOperation.java:191)
	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDurationOfInvocation(IOStatisticsBinding.java:464)
	at org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:189)
	at org.apache.hadoop.fs.azurebfs.services.AbfsClient.renamePath(AbfsClient.java:509)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.rename(AzureBlobFileSystemStore.java:986)
	at org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.rename(AzureBlobFileSystem.java:456)
	at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl.mvWithinFileSystem(MSFsUtilsImpl.scala:147)
	at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl.$anonfun$mv$2(MSFsUtilsImpl.scala:355)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl.fsTSG(MSFsUtilsImpl.scala:223)
	at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl.$anonfun$mv$1(MSFsUtilsImpl.scala:331)
	at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
	at com.microsoft.spark.notebook.common.trident.CertifiedTelemetryUtils$.withTelemetry(CertifiedTelemetryUtils.scala:71)
	at com.microsoft.spark.notebook.msutils.impl.MSFsUtilsImpl.mv(MSFsUtilsImpl.scala:331)
	at mssparkutils.IFs.mv(fs.scala:24)
	at mssparkutils.IFs.mv$(fs.scala:20)
	at notebookutils.fs$.mv(utils.scala:12)
	at notebookutils.fs.mv(utils.scala)
	at jdk.internal.reflect.GeneratedMethodAccessor218.invoke(Unknown Source)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244)
	at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374)
	at py4j.Gateway.invoke(Gateway.java:282)
	at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132)
	at py4j.commands.CallCommand.execute(CallCommand.java:79)
	at py4j.GatewayConnection.run(GatewayConnection.java:238)
	at java.base/java.lang.Thread.run(Thread.java:829)

[v-shex-msft]

Hi @cailen_cg ,

Any update on this? Did the above suggestions help with your scenario? if that is the case, you can consider Kudo or Accept the helpful suggestions to help others who faced similar requirements.

If these also don't help, please share more detailed information and description to help us clarify your scenario to test.

How to Get Your Question Answered Quickly 

Regards,

Xiaoxin Sheng

[v-shex-msft · Accepted solution]

Hi @cailen_cg,

I test with your code and find it create a file in Files folder and name with processed.

For this scenario, I'd like to suggest you add the filename to the 'destination dir' parameter and try again with following code:

from notebookutils import mssparkutils

filename='test.json'
Source_path = 'Files/' + filename
destination_dir = 'Files/processed/' + filename
 
notebookutils.fs.mv(Source_path, destination_dir, True)

Regards,

Xiaoxin Sheng