Forum Discussion

ArtecX's avatar
ArtecX
Frequent Visitor
2 years ago

Problem with date colum when staging

Have anyone else had problem with date colum when using staging mode on Dataflow Gen 2? My dataflow have started getting error "the type of column 'date' doesn't match the type of the column in the ...
Dataflows
miguel's avatar
miguel
Community Admin
2 years ago

Staging is orthogonal to where the destination should be. You can have the staging mechanism, which happens internally as part of the Dataflow Gen2 process to the staging artifact, but you will load your data to your own Lakehouse.

 

Using the staging artifacts (lakehouse and warehouse) as final destinations for your data can create issues on how the staging mechanism and other mechanisms work.

Again, we do not encourage the usage of the system artifacts for anything other than internal / system processes that Dataflow Gen2 uses.

 

The way that the staging mechanism works is more as part of the query that you create and if the query has "staging" enabled or not. There are temporal tables created for the staging mechanism at the system artifact level

 

Date and DateTime are supported datatypes for the staging artifact. However, you can see the full list of supported data types per destination on the link below:

Data Factory Dataflow Gen2 limitations - Microsoft Fabric | Microsoft Learn 

ArtecX's avatar
ArtecX
Frequent Visitor
2 years ago

Maybe i wasnt precise enough.

 

Im not using the system staging table for anything i just checked it out in Azure Data studio to troubleshoot the issue.

 

I have a dataflow gen 2 . where i have 9 queries. I have a Fabric Datawarehouse wich is the destination for the queries. «This is a self created Fabric Datawarehouse»

 

I have 2 queries that has staging enabled

 

There is a bug with the system artifact so the dataflow gen 2 staging table(System artifact staging table, from the staginglakehouse hidden from workspace) have somehow staged my date colum as date/time instead of date , so when i try to run the querie and it tries to stage it says the collum in the destination(StagingLakehouse) is a diffrent type.


  • miguel's avatar
    miguel
    Community Admin
    2 years ago

    Do you think that you could share some repro steps so that we could test the scenario on our end? If you could share any sample code that you're using and we could replicate on our end that would help tremendously 

    • v-cboorla-msft's avatar
      v-cboorla-msft
      Microsoft Employee
      2 years ago

      Hi ArtecX 

       

      We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
      In case if you have any resolution please do share that same with the community as it can be helpful to others.
      Otherwise, will respond back with the more details and we will try to help.


      Thanks

      • v-cboorla-msft's avatar
        v-cboorla-msft
        Microsoft Employee
        2 years ago

        Hi ArtecX 


        We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
        In case if you have any resolution please do share that same with the community as it can be helpful to others.
        If you have any question relating to the current thread, please do let us know and we will try out best to help you.
        In case if you have any other question on a different issue, we request you to open a new thread.


        Thanks