Writing Base64 string as file into Onelake /Files/<folder>

I managed to get it functioning because the user account running the notebook had complete access to the /Files directory. Since this was a one-time task, I didn't proceed to integrate it with a Service Principal or Managed Identity.

The following code worked well for me:

collected_data = df_with_decoded_data.collect()

base_dir = "/lakehouse/default/Files/attachments"  # File API Path
# Ensure the base directory exists
os.makedirs(base_dir, exist_ok=True)

# Write the decoded bytes to the file
for item in collected_data:
    # Construct the filename using AttachmentId and FileName
    filename = item["AttachmentId"] + item["FileName"][-4:]
    # Full path for the file
    file_path = os.path.join(base_dir, filename)

    # Ensure the directory for the file exists (in case the filename includes subdirectories)
    os.makedirs(os.path.dirname(file_path), exist_ok=True)

    # Write the Body content to the file
    with open(file_path, "wb") as file:
        file.write(item["DecodedData"])