Forum Discussion

arpost's avatar
arpost
Icon for Post Prodigy rankPost Prodigy
2 years ago
Solved

Does loading to Delta table from Files duplicate data in a lakehouse?

Greetings, community. I have a bunch of files I'm planning to load into a Lakehouse in CSV format. From there, I have considered loading them as Delta tables where possible. Does this duplicate the data, however, as the data is persisted in "raw" file format and then generated in Parquet format for the Delta table?

data engineering
Data Integration
one lake
  • AndyDDC's avatar
    AndyDDC
    2 years ago

    Hi arpost yes this will duplicate the data but you are transforming into a far better and more efficient format when saving as Delta, plus the underlying parquet will be compressed and likely smaller size than the source CSVs

1 Reply

  • AndyDDC's avatar
    AndyDDC
    Icon for Most Valuable Professional rankMost Valuable Professional
    2 years ago

    Hi arpost yes this will duplicate the data but you are transforming into a far better and more efficient format when saving as Delta, plus the underlying parquet will be compressed and likely smaller size than the source CSVs