Forum Discussion

arpost's avatar
arpost
Icon for Post Prodigy rankPost Prodigy
2 years ago
Solved

Does loading to Delta table from Files duplicate data in a lakehouse?

Greetings, community. I have a bunch of files I'm planning to load into a Lakehouse in CSV format. From there, I have considered loading them as Delta tables where possible. Does this duplicate the d...
data engineering
Data Integration
one lake
  • AndyDDC's avatar
    AndyDDC
    2 years ago

    Hi arpost yes this will duplicate the data but you are transforming into a far better and more efficient format when saving as Delta, plus the underlying parquet will be compressed and likely smaller size than the source CSVs

AndyDDC's avatar
AndyDDC
Icon for Most Valuable Professional rankMost Valuable Professional
2 years ago

Hi arpost yes this will duplicate the data but you are transforming into a far better and more efficient format when saving as Delta, plus the underlying parquet will be compressed and likely smaller size than the source CSVs