Forum Discussion

abhidotnet's avatar
abhidotnet
Icon for Advocate II rankAdvocate II
12 days ago
Solved

How do I unzip a .gz file?

I used a data pipeline to make a web call (http) and get a file. The file has been downloaded to the Files area in the lakehouse. How can I uncompress this file?  I am using a PySpark notebook t...
  • abhidotnet's avatar
    7 days ago

    Thanks.

    My issue was slightly different.

    My unzipping wasn't working because these were double zipped files. 

    This is the code I used:

    import gzip import os  # Local path to bronze folder bronze_local_path = "/lakehouse/default/Files/bronze"  # List .json.gz files gz_files = sorted([f for f in os.listdir(bronze_local_path) if f.endswith('.json.gz')])  print(f"Found {len(gz_files)} files to process\n")  extracted_json_files = []  for gz_file in gz_files:     gz_file_path = os.path.join(bronze_local_path, gz_file)     json_file_path = gz_file_path.replace('.json.gz', '.json')          print(f"Processing: {gz_file}")          try:         # Read and decompress         with open(gz_file_path, "rb") as f:             gz_data = f.read()                  decompressed = gzip.decompress(gz_data)                  # Check for double compression         if decompressed[:2] == b'\x1f\x8b':             print(f"  Detected double-compression. Decompressing again...")             decompressed = gzip.decompress(decompressed)                  # Write decompressed JSON locally         with open(json_file_path, "wb") as f:             f.write(decompressed)                  size_mb = len(decompressed) / (1024 ** 2)         print(f"  ✓ Decompressed: {size_mb:.2f} MB")                  extracted_json_files.append(json_file_path)              except Exception as e:         print(f"  ✗ Error: {str(e)}")  print(f"\n✓ Successfully extracted {len(extracted_json_files)} JSON files")