Debugging a complex dataflow

[DennesTorres]

Hi,

My dataflow is a bit complex, because I need to call an API, retrieve a JSON, work on the JSON to extract two different tables and materialize them in the lakehouse.

"->" means "calls"

Table A -> Function B -> Function C

Table D -> Function B -> Function C

Functions B and D have staging enabled with the silly hope it would avoid calling the API twice. I don't think it is.

Tables A and D have staging disable on the hope the processing would be faster and avoid the Compute stage.

The result is extremely weird:

Both tables, A and D, are correctly materialized in the lakehouse and report success. However, function C reports a conversion error from LIST to Text.

I understand the error message, it would not be a surprise by itself and it would require investigation on the JSON received. However, how is it possible the table materialization to report success and have the data in it while the source query for them reports an error?

While writting this message it came to my mind it could be the staging step. So, I disabled staging on B and C. 

Same result. It continues giving me a conversion error when creating a table on the function C, although the two resulting tables are created sucesfully.

Below are the screen capture of the execution history and the entire error message. I can't just accept the resulting tables and leave like this because some record may be lost in the middle of the process.

How do I debug the dataflow to find this error?

I already tried to use the option "Remove Rows -> Keep errors" with all columns selected, but it always return nothing.

"MainRecord: Error Code: 104100, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Expression.Error: Failed to insert a table., InnerException: We cannot convert a value of type List to type Text., Underlying error: We cannot convert a value of type List to type Text. Details: Reason = Expression.Error;Message = We cannot convert a value of type List to type Text.;Message.Format = We cannot convert a value of type #{0} to type #{1}.;Message.Parameters = {"List", "Text"};Microsoft.Data.Mashup.Error.Context = User (Request ID: 6da62349-dda5-4e6c-8873-85e7a62d9a43)." 

("MainRecord" is the actual name of Function C )

Mind how the Function C stays running for a long time after the materialization of the two final tables are already completed and reported success.

Kind Regards,

Dennes

[jwelch]

Please disable staging on the functions and see if that changes the behavior. Generally, you don't want to stage functions. I'd need more detail on what you are doing in the functions to determine if it's appropriate here.

[DennesTorres]

Hi,
I tried to disable staging on the functions, but nothing changed. It was like if the function was still using staging, even when disabled.

Kind Regards,

Dennes

[v-nikhilan-msft]

Hi @DennesTorres 
Apologies for the delay in response. 
Are you still facing this issue? If yes please let us know. 
We will revert back with more details and try to help.

[DennesTorres · Accepted solution]

Hi,

The problem is there, but became very low priority.

I realized the dataflow was calling the API and in this way it would not reach the desired result. 

I created a 2nd version of the dataflow, which instead of calling the API, received the JSON (API result) as parameter. This 2nd version works without the failure.

I can't explain that. But I also can't use the 2nd version because I can't pass parameters from pipelines to dataflows. 

Kind Regards,

Dennes

[v-nikhilan-msft]

Hi @DennesTorres 
Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.
Thanks