Forum Discussion

craigmday's avatar
craigmday
Icon for Helper I rankHelper I
1 year ago

Dataflow Gen 2: Incremental Refresh fails with error: The value does not support versioning

Hi all,   I am trying to configure incremental refresh on a data source that contains financial entries (general ledger entries). I am using a Dataflow Gen 2 and am publishing to a Fabric Lakehouse...
data warehouse
warehouse
BhaveshPatel's avatar
BhaveshPatel
Icon for Super User rankSuper User
1 year ago

Hi craigmday,

 

For Incremental Refresh In Dataflow Gen 2, Sometimes Dataflow Gen 2 is failing. The reason behind that the data is not integar column in Delta Lake. For that, You have to first apply data ( Full Load ) in Dataflow Gen 2. After that, convert it into lakehouse notebooks and after that use incremental refresh.  ( below is a sample dimdate notebooks incremental refresh). This will be successful. Here, You see Year column is an integar column in Delta Lake.

 

### Variables
LakehouseName = "default"
TableName = "dimdate"
ColName = "Year"
NumberOfibusToRemove = "1"


### Remove Old Orders
Reload = spark.sql("SELECT Max({0})-{1} as ReLoadValue FROM {2}.{3}".format(ColName,NumberOfibusToRemove,LakehouseName,TableName)).collect()
Reload = Reload[0].ReLoadValue
spark.sql("Delete from {0}.{1} where {2} > {3}".format(LakehouseName, TableName, ColName, Reload))