Forum Discussion
abhidotnet
Advocate II
12 days agoHow do I unzip a .gz file?
I used a data pipeline to make a web call (http) and get a file. The file has been downloaded to the Files area in the lakehouse. How can I uncompress this file? I am using a PySpark notebook t...
- 7 days ago
Thanks.
My issue was slightly different.
My unzipping wasn't working because these were double zipped files.
This is the code I used:import gzip import os # Local path to bronze folder bronze_local_path = "/lakehouse/default/Files/bronze" # List .json.gz files gz_files = sorted([f for f in os.listdir(bronze_local_path) if f.endswith('.json.gz')]) print(f"Found {len(gz_files)} files to process\n") extracted_json_files = [] for gz_file in gz_files: gz_file_path = os.path.join(bronze_local_path, gz_file) json_file_path = gz_file_path.replace('.json.gz', '.json') print(f"Processing: {gz_file}") try: # Read and decompress with open(gz_file_path, "rb") as f: gz_data = f.read() decompressed = gzip.decompress(gz_data) # Check for double compression if decompressed[:2] == b'\x1f\x8b': print(f" Detected double-compression. Decompressing again...") decompressed = gzip.decompress(decompressed) # Write decompressed JSON locally with open(json_file_path, "wb") as f: f.write(decompressed) size_mb = len(decompressed) / (1024 ** 2) print(f" ✓ Decompressed: {size_mb:.2f} MB") extracted_json_files.append(json_file_path) except Exception as e: print(f" ✗ Error: {str(e)}") print(f"\n✓ Successfully extracted {len(extracted_json_files)} JSON files")
tayloramy
Super User
11 days agoHi abhidotnet,
You can use the gzip package in Python to unzip a .gz file.
import gzip
import shutil# Define paths (e.g., in your Lakehouse Files section)
input_path = "/lakehouse/default/Files/my_data.csv.gz"
output_path = "/lakehouse/default/Files/my_data.csv"# Decompress the .gz file
with gzip.open(input_path, "rb") as f_in:
with open(output_path, "wb") as f_out:
shutil.copyfileobj(f_in, f_out)
print(f"Successfully unzipped to {output_path}")