Create additional column for partition id

[coolie]

I am reading Rest data to create a lakehouse table, but I'd now like the table to be partitioned by date of the daily run. I eventually managed to add the additional source column (derived from pipeline run time) as a string, and add it to the mapping, but get an error when I run the pipeline: 

"ErrorCode=PartitioneddestinationMissingTypeConversion

Message=destination partition required input TypeConversion is missing"

Additional columns have to be strings, and strings are permitted for partitioning, so not sure what I can do to fix it?

[v-cboorla-msft]

Hi @coolie 

Thanks for using Fabric Community.

Could you please give us more details about the additional source column which you have enabled partition for? Please provide some screen shots of the copy data mapping. This would help me to understand the issue and help you better.

Thanks

[coolie]

These are the additional columns. I have 2 variables. The first is pipeline trigger time in format yyyyDDmm. The second is productId created from a variable.

Source:

Destination:

And here is the mapping. I've also tried long as the target type:

I am currently using overwrite because if I use append you get duplicate data if the pipeline is run more than once, as there does not seem to be an upsert option. Im hoping that once I get partitioning working, it will only overwrite the relevant partition and not the whole lakehouse (see https://community.fabric.microsoft.com/t5/Data-Pipelines/Appending-data-to-a-partitioned-table-in-th... )

[v-cboorla-msft]

Hi @coolie 

Thanks for your reponse.

I have tried to repro the same scenario but couldn't find any error. So could you please confirm if you are missing something or if I am missing any steps in the process.

Please find the screenshots and the steps I followed for my repro.

To do the repro I took source as Lakehouse and selected the sampledata table, destination as Lakeshouse named the output as DemoTable.

Source:

Destination:

Mapping:

Result:

I hope this helps. If you have any further questions please do let us know.

[coolie]

Hi thanks for taking the time to do that. I also reproduced what you did successfully. The difference is my source is a REST (external) connection. If you keep everything the same just change the source you will find you get the same error, so it must be a bug. I got round this by using a intermediate temp table before writing to the partitioned one. Unfortunately you cant append to a partitioned table so going to raise BUG #2. Not looking too good for Fabric so far is it.

[v-cboorla-msft]

Hi @coolie 

Apologies for the inconvenience that you are facing here.

This might require a deeper investigation from our engineering team about your workspace, the pipeline details and the logic behind it to properly understand what might be happening. 

Please go ahead and raise a support ticket to reach our support team:

https://support.fabric.microsoft.com/support
Please provide the ticket number here as we can keep an eye on it.

Thank you.

[v-cboorla-msft]

Hi @coolie 

We haven’t heard from you on the last response and was just checking back to see if you got a chance to create a support ticket. If yes please share the ticket number.
Otherwise, will respond back with the more details and we will try to help.

Thanks

[coolie]

Support request is 2312040050001409