Forum Discussion

coolie's avatar
coolie
Icon for Helper I rankHelper I
2 years ago

Create additional column for partition id

I am reading Rest data to create a lakehouse table, but I'd now like the table to be partitioned by date of the daily run. I eventually managed to add the additional source column (derived from pipel...
pipelines
v-cboorla-msft's avatar
v-cboorla-msft
Icon for Microsoft Employee rankMicrosoft Employee
2 years ago

Hi coolie 

 

Thanks for using Fabric Community.

Could you please give us more details about the additional source column which you have enabled partition for? Please provide some screen shots of the copy data mapping. This would help me to understand the issue and help you better.

 

Thanks

  • coolie's avatar
    coolie
    Icon for Helper I rankHelper I
    2 years ago

    These are the additional columns. I have 2 variables. The first is pipeline trigger time in format yyyyDDmm. The second is productId created from a variable.

    Source:

    Destination:

    And here is the mapping. I've also tried long as the target type:

    I am currently using overwrite because if I use append you get duplicate data if the pipeline is run more than once, as there does not seem to be an upsert option. Im hoping that once I get partitioning working, it will only overwrite the relevant partition and not the whole lakehouse (see https://community.fabric.microsoft.com/t5/Data-Pipelines/Appending-data-to-a-partitioned-table-in-the-Lakehouse/m-p/3396450 )

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Icon for Microsoft Employee rankMicrosoft Employee
      2 years ago

      Hi coolie 

       

      Thanks for your reponse.

       

      I have tried to repro the same scenario but couldn't find any error. So could you please confirm if you are missing something or if I am missing any steps in the process.

       

      Please find the screenshots and the steps I followed for my repro.

      To do the repro I took source as Lakehouse and selected the sampledata table, destination as Lakeshouse named the output as DemoTable.

      Source:

      Destination:

      Mapping:

      Result:

       

      I hope this helps. If you have any further questions please do let us know.

       

      • coolie's avatar
        coolie
        Icon for Helper I rankHelper I
        2 years ago

        Hi thanks for taking the time to do that. I also reproduced what you did successfully. The difference is my source is a REST (external) connection. If you keep everything the same just change the source you will find you get the same error, so it must be a bug. I got round this by using a intermediate temp table before writing to the partitioned one. Unfortunately you cant append to a partitioned table so going to raise BUG #2. Not looking too good for Fabric so far is it.