Create additional column for partition id

Question

I am reading Rest data to create a lakehouse table, but I'd now like the table to be partitioned by date of the daily run. I eventually managed to add the additional source column (derived from pipeline run time) as a string, and add it to the mapping, but get an error when I run the pipeline:

"ErrorCode=PartitioneddestinationMissingTypeConversion

Message=destination partition required input TypeConversion is missing"

Additional columns have to be strings, and strings are permitted for partitioning, so not sure what I can do to fix it?

coolie · Answer

Support request is&nbsp;2312040050001409

v-cboorla-msft · Answer

Hi&nbsp;coolie&nbsp;
&nbsp;
Thanks for using Fabric Community.
Could you please give us more details about the additional source column which you have enabled partition for? Please provide some screen shots of the copy data mapping. This would help me to understand the issue and help you better.
&nbsp;
Thanks

coolie · Answer

These are the additional columns. I have 2 variables. The first is pipeline trigger time in format yyyyDDmm. The second is productId created from a variable.

Source:

Destination:

And here is the mapping. I've also tried long as the target type:

I am currently using overwrite because if I use append you get duplicate data if the pipeline is run more than once, as there does not seem to be an upsert option. Im hoping that once I get partitioning working, it will only overwrite the relevant partition and not the whole lakehouse (see https://community.fabric.microsoft.com/t5/Data-Pipelines/Appending-data-to-a-partitioned-table-in-the-Lakehouse/m-p/3396450 )

v-cboorla-msft · Answer

Hi&nbsp;coolie&nbsp;
&nbsp;
Thanks for your reponse.
&nbsp;
I have tried to repro the same scenario but couldn't find any error. So could you please confirm if you are missing something or if I am missing any steps in the process.
&nbsp;
Please find the screenshots and the steps I followed for my repro.
To do the repro I took source as Lakehouse and selected the sampledata table, destination as Lakeshouse named the output as DemoTable.
Source:

Destination:

Mapping:

Result:

&nbsp;
I hope this helps. If you have any further questions please do let us know.
&nbsp;

coolie · Answer

Hi thanks for taking the time to do that. I also reproduced what you did successfully. The difference is my source is a REST (external) connection. If you keep everything the same just change the source you will find you get the same error, so it must be a bug. I got round this by using a intermediate temp table before writing to the partitioned one. Unfortunately you cant append to a partitioned table so going to raise BUG #2. Not looking too good for Fabric so far is it.