Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Register now to learn Fabric in free live sessions led by the best Microsoft experts. From Apr 16 to May 9, in English and Spanish.

Reply
DennesTorres
Solution Supplier
Solution Supplier

Debugging a complex dataflow

Hi,

My dataflow is a bit complex, because I need to call an API, retrieve a JSON, work on the JSON to extract two different tables and materialize them in the lakehouse.

 

"->" means "calls"


Table A -> Function B -> Function C

Table D -> Function B -> Function C

 

Functions B and D have staging enabled with the silly hope it would avoid calling the API twice. I don't think it is.

 

Tables A and D have staging disable on the hope the processing would be faster and avoid the Compute stage.

 

The result is extremely weird:

Both tables, A and D, are correctly materialized in the lakehouse and report success. However, function C reports a conversion error from LIST to Text.

I understand the error message, it would not be a surprise by itself and it would require investigation on the JSON received. However, how is it possible the table materialization to report success and have the data in it while the source query for them reports an error?

While writting this message it came to my mind it could be the staging step. So, I disabled staging on B and C. 

Same result. It continues giving me a conversion error when creating a table on the function C, although the two resulting tables are created sucesfully.

 

Below are the screen capture of the execution history and the entire error message. I can't just accept the resulting tables and leave like this because some record may be lost in the middle of the process.

How do I debug the dataflow to find this error?

I already tried to use the option "Remove Rows -> Keep errors" with all columns selected, but it always return nothing.

"MainRecord: Error Code: 104100, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: Expression.Error: Failed to insert a table., InnerException: We cannot convert a value of type List to type Text., Underlying error: We cannot convert a value of type List to type Text. Details: Reason = Expression.Error;Message = We cannot convert a value of type List to type Text.;Message.Format = We cannot convert a value of type #{0} to type #{1}.;Message.Parameters = {"List", "Text"};Microsoft.Data.Mashup.Error.Context = User (Request ID: 6da62349-dda5-4e6c-8873-85e7a62d9a43)." 

("MainRecord" is the actual name of Function C )

DennesTorres_0-1707842007481.png


Mind how the Function C stays running for a long time after the materialization of the two final tables are already completed and reported success.

 

Kind Regards,

 

Dennes

1 ACCEPTED SOLUTION

Hi,

The problem is there, but became very low priority.

I realized the dataflow was calling the API and in this way it would not reach the desired result. 

I created a 2nd version of the dataflow, which instead of calling the API, received the JSON (API result) as parameter. This 2nd version works without the failure.

I can't explain that. But I also can't use the 2nd version because I can't pass parameters from pipelines to dataflows. 

Kind Regards,

 

Dennes

View solution in original post

5 REPLIES 5
jwelch
Employee
Employee

Please disable staging on the functions and see if that changes the behavior. Generally, you don't want to stage functions. I'd need more detail on what you are doing in the functions to determine if it's appropriate here.

Hi,
I tried to disable staging on the functions, but nothing changed. It was like if the function was still using staging, even when disabled.

 

Kind Regards,

 

Dennes

Hi @DennesTorres 
Apologies for the delay in response. 
Are you still facing this issue? If yes please let us know. 
We will revert back with more details and try to help.

Hi,

The problem is there, but became very low priority.

I realized the dataflow was calling the API and in this way it would not reach the desired result. 

I created a 2nd version of the dataflow, which instead of calling the API, received the JSON (API result) as parameter. This 2nd version works without the failure.

I can't explain that. But I also can't use the 2nd version because I can't pass parameters from pipelines to dataflows. 

Kind Regards,

 

Dennes

v-nikhilan-msft
Community Support
Community Support

Hi @DennesTorres 
Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.
Thanks

Helpful resources

Announcements
Microsoft Fabric Learn Together

Microsoft Fabric Learn Together

Covering the world! 9:00-10:30 AM Sydney, 4:00-5:30 PM CET (Paris/Berlin), 7:00-8:30 PM Mexico City

March 2024 FBC Gallery Image

Fabric Monthly Update - March 2024

Check out the March 2024 Fabric update to learn about new features.

April Fabric Community Update

Fabric Community Update - April 2024

Find out what's new and trending in the Fabric Community.

Top Solution Authors