Forum Discussion
Problem with date colum when staging
I havent tried to create a new datawarehouse, since thats not an option. The staging artifact is hidden from Workspace view, but you are still able to browse it through azure data studio. If i remove staging the dataflow gen 2 works. but my problem is that the dataflow gen 2 does take 7 hours longer when staging is off versus when its on for specific queries.
Staging is orthogonal to where the destination should be. You can have the staging mechanism, which happens internally as part of the Dataflow Gen2 process to the staging artifact, but you will load your data to your own Lakehouse.
Using the staging artifacts (lakehouse and warehouse) as final destinations for your data can create issues on how the staging mechanism and other mechanisms work.
Again, we do not encourage the usage of the system artifacts for anything other than internal / system processes that Dataflow Gen2 uses.
The way that the staging mechanism works is more as part of the query that you create and if the query has "staging" enabled or not. There are temporal tables created for the staging mechanism at the system artifact level
Date and DateTime are supported datatypes for the staging artifact. However, you can see the full list of supported data types per destination on the link below:
Data Factory Dataflow Gen2 limitations - Microsoft Fabric | Microsoft Learn
Maybe i wasnt precise enough.
Im not using the system staging table for anything i just checked it out in Azure Data studio to troubleshoot the issue.
I have a dataflow gen 2 . where i have 9 queries. I have a Fabric Datawarehouse wich is the destination for the queries. «This is a self created Fabric Datawarehouse»
I have 2 queries that has staging enabled
There is a bug with the system artifact so the dataflow gen 2 staging table(System artifact staging table, from the staginglakehouse hidden from workspace) have somehow staged my date colum as date/time instead of date , so when i try to run the querie and it tries to stage it says the collum in the destination(StagingLakehouse) is a diffrent type.
Do you think that you could share some repro steps so that we could test the scenario on our end? If you could share any sample code that you're using and we could replicate on our end that would help tremendously
Hi ArtecX
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet.
In case if you have any resolution please do share that same with the community as it can be helpful to others.
Otherwise, will respond back with the more details and we will try to help.
Thanks
