Forum Discussion

Anonymous's avatar
Anonymous
Not applicable
2 years ago
Solved

mssparkutils.fs.mv is giving : java.io.IOException: when trying to move files

Hi All,

 

I am trying to move files from one a workspace in one subscription to another workspace in another subscription using mssparkutils.fs.mv. mssparkutils.fs.cp is working file. I am able to copy files but move is giving error.

 

We have created trusted workspace between the subscription. Also created shortcut for second workspace. Below is the code that I am using - 

import os
for file in mssparkutils.fs.ls(scripdeltapath):  
    updated_path = os.path.join(stagingdeltapath, file.name) 
    print(f"Moving from {file.path} -> {updated_path}")
    try:
        mssparkutils.fs.cp(file.path,updated_path)
    except Exception as e:
           print(str(e))
        mssparkutils.notebook.exit("Error: Something went wrong!", status="error")

 

Let me know if you need more infromation.

 

Thanks,

Deepmala

data engineering
data pipeline
notebook
  • AndyDDC's avatar
    AndyDDC
    2 years ago

    Hi Anonymous have you tried adding the parameters that create the destinations directory?

     

    mssparkutils.fs.mv(file.path,updated_path,True,True)
     
     
    mssparkutils.fs.mv('source file or directory', 'destination directory', True) # Set the last parameter as True to firstly create the parent directory if it does not exist

    mssparkutils.fs.mv('source file or directory', 'destination directory', True, True) # Set the third parameter to True to firstly create the parent directory if it does not exist. Set the last parameter to True to overwrite the updates.

3 Replies

  • AndyDDC's avatar
    AndyDDC
    Most Valuable Professional
    2 years ago

    Hi Anonymous have you tried adding the parameters that create the destinations directory?

     

    mssparkutils.fs.mv(file.path,updated_path,True,True)
     
     
    mssparkutils.fs.mv('source file or directory', 'destination directory', True) # Set the last parameter as True to firstly create the parent directory if it does not exist

    mssparkutils.fs.mv('source file or directory', 'destination directory', True, True) # Set the third parameter to True to firstly create the parent directory if it does not exist. Set the last parameter to True to overwrite the updates.
  • Anonymous's avatar
    Anonymous
    Not applicable
    2 years ago

    Hi Anonymous 

     

    I used your code to move files from one workspace to another workspace without experiencing any error. I tried mssparkutils.fs.mv and mssparkutils.fs.cp, both worked. What error did you get when moving files? Is there any error message?

    In addition, if you move files to a workspace which locates in the same Fabric Capacity as the source Workspace does, will it succeed?

     

    More information: in my testing, I was using ABFS paths and moving files across workspaces within the same Fabric Trial Capacity. 

     

    Best Regards,
    Jing

    • Anonymous's avatar
      Anonymous
      Not applicable
      2 years ago

      Hi,

       

      For me as well, cp worked but mv was giving IOException error as mentioned in my post. Also I have created shortcuts and by using that shortcuts I was trying to move files from workspace to another in two different subcriptions.