Forum Discussion

Kaatiiaa's avatar
Kaatiiaa
Helper I
2 years ago
Solved

DataFlow Incremental Refresh

Im trying to incrementally append data in Lakehouse using DataFlow (as described here: https://learn.microsoft.com/en-us/fabric/data-factory/tutorial-setup-incremental-refresh-with-dataflows-gen2)

I did everything in the same way as described.

If I open DataFlow I can see in preview that it works as my main query/table is filtered according to the filter based on maximum order_id (the result of the second query).

The problem is when I want to refresh the dataflow. I get an error message "We cannot use operator < for Table and Number types"

Any idea what is wrong? I guess that if I can see in preview that the table is filtered according to maximum order_id so data type is ok (number). The filter is set exactly as in the example described on LearnMicrosoft site.

Thank you

 

The whole text of error message:

Error Code: Mashup Exception Expression Error, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Cannot insert table., Inner exception: We cannot use operator < for Table and Number types., basic error: We cannot use operator < for Table and Number types. Details: Reason = Expression.Error;Message = We cannot use the < operator for Table and Number types.;Detail = [Operator = "<", Left = error "Microsoft.Mashup.Engine1.Runtime.ValueException: [Expression.Error] No value specified.#(cr)#(lf) at Microsoft.Mashup.Engine1.Language.ValueCreator.CreateValueForThrow(IThrowExpression throwExpr)#(cr)#(lf) at Microsoft.Mashup.Engine1.Language.ValueCreator.<>c__DisplayClass23_0.<CreateValueForRecord>b__0(Int32 index)#(cr)#(lf) at Microsoft.Mashup.Engine1.Runtime.RecordValue.DemandRecordValue.get_Item(Int32 index)#(cr)#(lf) at Microsoft.Data.Mashup.ProviderCommon.MashupResource.TryGetValue(Func`1 getValue, IValue& value, String& errorMessage)#(cr)#(lf)Record", Right = 2619 65];Message.Format = We cannot use the #{0} operator for types #{1} and #{2}.;Message.Parameters = {"<", "Table", "Number"};Microsoft.Data.Mashup.Error.Context = User GatewayObjectId: 4c147d88-9572-4e34-8a5b-4fe1391a873e (Request ID: 1318301e-5b5c-44a4-9ea9-30d02261f7e1).

 

 

Dataflow
  • frithjof_v's avatar
    frithjof_v
    2 years ago

    The setup seems good to me.

     

    "refreshing the table using Replace method works fine."

     

    Are you saying this exactly same dataflow runs successfully if you choose Replace method, it only fails if you choose Append method?

     

    So your M code is exactly the same in both cases. The only change is the destination settings?

     

    So the issue seems to be related to the destination settings?

     

    However, using Replace method will erase all the old data so you don't want to use it in your case. You want to Append to keep the old data and add new data.

     

     

    Btw, I see there are a couple of known issues with Data Factory <> Oracle.

    https://learn.microsoft.com/en-us/fabric/get-started/fabric-known-issues

     

    Could the issue be relates to the number format? Could you try to force the number format to whole number (Int64)?

     

    (Please do the testing in a test workspace in your Fabric, not production workspace.)

     

    Tbh I'm confused why you are getting an error.

     

    Just to make sure, the IncrementalEventID query has disabled staging, and this query does not have a destination, right?

     

    Perhaps you could create a support ticket to have the support team look at the case.

11 Replies