mssparkutils file operations
I am having difficulty moving files from one location to another using mssparkutils in a fabric notebook. The error seems to be random as sometimes the commands execute, other times it returns an error. I am moving files from one location in a shortcut to an ADLS Gen 2 storage account to another location in the same account. I am using the full "abfss://.." file path for the source and destination folder. I have confirmed that the documentation for msssparkutils.fs.mv is incorrect. Using param createPath generates an error that says parameter createPath is undefined. However, putting a value of True does make the function work, however the if the destination folder does not exist it is NOT created but rather generates an error. I have worked around this my using the mkdirs command each time as it does not return an error if the directory already exists. Thes commands are being issued in a loop as I am ingesting all csv files in a folder and moving them to another folder after each is ingested. Periodically the mkdirs or mv command generates an error from what appears to be a permissions issue that I can not figure out how to resolve. It seems random. As an example this is the 5th time through the loop when it throws the below error: (specifics xxx for privacy) If it matters the Storage account shortcut was created using my credentials and I am the one running the notebook. Failed in processing data from raw folder with exception:An error occurred while calling z:mssparkutils.fs.mkdirs. : java.nio.file.AccessDeniedException: Operation failed: "Forbidden", 403, PUT, "http://onelake.dfs.fabric.microsoft.com/xxxFiles/xxx/2024-01-27?resource=directory&timeout=90," AuthorizationPermissionMismatch, "This request is not authorized to perform this operation using this permission. RequestId:1ce9a06f-001f-0003-594f-515e63000000 Time:2024-01-27T18:31:55.5350440Z"