Forum Discussion
Impactful IndividualLanding zone in OneLake
We'd like to load data from source systems into a landing zone first, before merging it with our Bronze layer. All in OneLake of course.
If we try to do that, choosing overwrite as load option, it creates duplicate parquet files every time. That is not necessary in this case and will costs a huge amount of storage space over time. There does not seem to be a way to import the data in a "truncate and load and do not keep history" way.
What would nbe your suggestion to do for this?
Many thanks.
3 Replies
sudhavHelper V
I guess, along with the lakehouse there is one staging lake house also creating, so first your data will land there and go to original lake house. i am not an expert, just its my thought.
ajaroraMicrosoft Employee
Walther, Are you using pipelines copy activity to load data ?
If so, can you please explain further what do you mean by duplicate parquet files left around ? Are you compacting/vacuuming the destination table in lakehouse for the unused files ? If so let me know we may have something for you.
DennesTorresPower Participant
Hi,
You are right, when using the lakehouse "Files" as destination, this option is really missing.
To reach this result, you would need to drop the folder, if it already exists, before transfering content to your landing zone.
In my personal opinion, I don't like this that much. By joining the data directly to your Bronze, you lose data lineage. The landing zone would be a good place to keep the blocks of data loaded together and in this way you would know exactly what records or values where loaded each day. To achieve this, your landing zone needs to keep one forlder for each ingestion. Of course, it's a choice, I prefer this design to alway keep track of everything and the storage is not so expensive.
Kind Regards,Dennes