- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Printer Friendly Page
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Appending data to a partitioned table in the Lakehouse
I have been able to create a partitioned table in a Lakehouse usign a pipeline where the data is read from a CSV file and partitioned on the YEAR_MONTH field (all data in the file is of the same YEAR_MONTH), and then sub-partitioned on another two fields in the file. However, as new data is realeased on a monthly basis I want to be able to append it to this table. At the moment this is not possible using the same method as the original table was created. It seems you can paritition a table when using OVERWRITE but not when using APPEND (even though adding the new data will create an additional top level parition and not add data to any of the exisiting partitions).
I am missing something obvious? Is this a preview restriction? Are there other ways of achieving the same thing?
Solved! Go to Solution.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I made it work. Is there a better solution to this, very likely.
First, I created a regular table using the overwrite option in ADF
2nd, I created an empty delta-partitioned table in a notebook
3rd I inserted into the partitioned table from the staging table created in the first step.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Sorry I missed the last screenshot
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks, I will try this out today. 🙂
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I made it work. Is there a better solution to this, very likely.
First, I created a regular table using the overwrite option in ADF
2nd, I created an empty delta-partitioned table in a notebook
3rd I inserted into the partitioned table from the staging table created in the first step.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
The original table was created in the lakehouse with the following parameters
But when you select the append option the fields for enabling paritioning disappear and the execution of the pipeline fails.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for the info.
I'll try this myself, but I wonder if the overwrite partition clause is just to define the table.
Once you have to append, the table is already defined. You just go ahead and append. Data should flow into the proper partitions
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Agreed, that was my expectation but, as I say, it errors when run.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Oh well, I was not expecting this:
I am starting to think that our only option today is to go to a staging table with ADF and they do the insert into the partitioned using a notebook script.
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
May I ask you how you created the Delta Lake table? Directly from a Copy? Can you share a screenshot?

Helpful resources
Join our Fabric User Panel
This is your chance to engage directly with the engineering team behind Fabric and Power BI. Share your experiences and shape the future.
Fabric Monthly Update - June 2025
Check out the June 2025 Fabric update to learn about new features.

Subject | Author | Posted | |
---|---|---|---|
04-10-2025 07:06 AM | |||
07-14-2023 02:11 AM | |||
12-17-2024 06:29 AM | |||
04-23-2025 08:28 AM | |||
09-02-2024 04:20 AM |
User | Count |
---|---|
2 | |
2 | |
1 | |
1 |
User | Count |
---|---|
2 | |
2 | |
2 | |
2 | |
2 |