Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Earn a 50% discount on the DP-600 certification exam by completing the Fabric 30 Days to Learn It challenge.

Reply
mkcongrove1
Advocate I
Advocate I

Has anyone gotten Incremental Refresh to work with only date?

All of the articles, and all of my searching tell me no - that I have to have my source data in DateTime. But wanted to see if there was some hidden knowledge that could be shared 🙂

 

My current hurdle is that I have multi-billion record tables that store date as a decimal (I'm not IT, don't ask me why). I can convert this to Integer easily. However, I'm afraid my database doesn't have the capacity to convert these billions of records to DateTime format, from their current Decimal format - the transformation will fail due to having to convert to text first and then to DateTime (I have yet to figure out how "Query folding DEFINITELY works with Snowflake", but that's a different issue).

 

I was successful in setting up the parameters and function to work with my date, but I run into my roadblock when I go to "Define Parameters" after importing. 

mkcongrove1_0-1706799080892.png

I know the error is because I don't have things in DateTime - so, has anyone figured out how to work around this issue?

 

1 ACCEPTED SOLUTION
DataInsights
Super User
Super User

@mkcongrove1,

 

Incremental refresh does require a DateTime column. I've accomplished this with custom SQL and Snowflake functions such as TO_TIMESTAMP and TIMESTAMP_FROM_PARTS. Once you convert the decimal to integer, you can use TIMESTAMP_FROM_PARTS and SUBSTRING to construct a DateTime column. You might explore the most performant options for the DateTime conversion (Snowflake view, function, etc.).





Did I answer your question? Mark my post as a solution!

Proud to be a Super User!




View solution in original post

1 REPLY 1
DataInsights
Super User
Super User

@mkcongrove1,

 

Incremental refresh does require a DateTime column. I've accomplished this with custom SQL and Snowflake functions such as TO_TIMESTAMP and TIMESTAMP_FROM_PARTS. Once you convert the decimal to integer, you can use TIMESTAMP_FROM_PARTS and SUBSTRING to construct a DateTime column. You might explore the most performant options for the DateTime conversion (Snowflake view, function, etc.).





Did I answer your question? Mark my post as a solution!

Proud to be a Super User!




Helpful resources

Announcements
LearnSurvey

Fabric certifications survey

Certification feedback opportunity for the community.