Forum Discussion
Progrmatically write files in delta
I was able to use this code to write to a simple text file:
import os # Define the sentence you want to write sentence = "This is the sentence that will be written to the text file." # Specify the folder path and file name folder_base_path = "/lakehouse/default/Files/" folder_relative_path = "sentence_files" file_name = "output.txt" folder_path = os.path.join(folder_base_path, folder_relative_path) # Combine the folder path and file name file_path = os.path.join(folder_path, file_name) # Create the directory if it doesn't exist os.makedirs(folder_path, exist_ok=True) # Open the file in write mode and write the sentence with open(file_path, "w") as file: file.write(sentence) print(f"Sentence written to {file_path}")The folder_base_path will depend on whether your notebook has a default lakehouse or if you are just mounting lakehouses to your notebook.
In the code I show above, the folder_base_path assumes that the notebook has a default lakehouse.
If you don't want to use a default lakehouse, then you will need to mount a lakehouse instead. However if you don't have any specific requirements, I would say just use a default lakehouse for your notebook.
https://fabric.guru/how-to-mount-a-lakehouse-and-identify-the-mounted-lakehouse-in-fabric-notebook
By default, PySpark creates a folder with multiple files. I guess this is because PySpark uses distributed processing on multiple worker nodes.
If you want to write a (not too big) dataframe to a single file, I think the easiest way is to use Pandas.
I was able to use this code to write to a simple text file:
import os
# Define the sentence you want to write
sentence = "This is the sentence that will be written to the text file."
# Specify the folder path and file name
folder_base_path = "/lakehouse/default/Files/"
folder_relative_path = "sentence_files"
file_name = "output.txt"
folder_path = os.path.join(folder_base_path, folder_relative_path)
# Combine the folder path and file name
file_path = os.path.join(folder_path, file_name)
# Create the directory if it doesn't exist
os.makedirs(folder_path, exist_ok=True)
# Open the file in write mode and write the sentence
with open(file_path, "w") as file:
file.write(sentence)
print(f"Sentence written to {file_path}")
The folder_base_path will depend on whether your notebook has a default lakehouse or if you are just mounting lakehouses to your notebook.
In the code I show above, the folder_base_path assumes that the notebook has a default lakehouse.
If you don't want to use a default lakehouse, then you will need to mount a lakehouse instead. However if you don't have any specific requirements, I would say just use a default lakehouse for your notebook.
https://fabric.guru/how-to-mount-a-lakehouse-and-identify-the-mounted-lakehouse-in-fabric-notebook
By default, PySpark creates a folder with multiple files. I guess this is because PySpark uses distributed processing on multiple worker nodes.
If you want to write a (not too big) dataframe to a single file, I think the easiest way is to use Pandas.
I usually write to delta tables and not to files. Write to tables create the table with exact same desiredTblName. I was hoping for the same to happen when it comes to files. But to my surprise, it creates a folder in the files section with desiredName (that I was hoping to be the file name) and the actual file name is spark generated.
I have also observed that if you are manual uploading a file / using pipeline to copy a parquet (or any other available format) file to a sink/destination, the system writes it with exact same desiredName.
The file name control is an important aspect of my workflow, hence I cant stick writing to files using codes in notebook. I don't know if there is a lakehouse api that lets you write(upload) a file with developer created content such that the file name could be exactly same as dev desired. If it happens, I want to try it out, but for now I have broken my code to let pipeline handle that part.