Data getting lost between sequential dataflows

[brft]

To increase performance when analyzing  a large on-prem Excel file, I split a single dataflow into three sequential dataflows.  Performance has increased substantially, but I am seeing some missing data in a particular table I can't understand.

My lineage:

Staging generates 3 tables.  One of these is called "Notes", and it is the nearly raw data from one tab of the excel file.  This dataflow runs without issue, and I see the following information in the refresh history (consistent):

Requested on

Dataflow name

Dataflow refresh status

Table name

Partition name

Refresh status

Start time

End time

Duration

Rows processed

Bytes processed (KB)

Max commit (KB)

Processor Time

Wait time

Compute engine

Error

26-07-2022 16:35

File Staging Dataflow

Completed

Notes

FullRefreshPolicyPartition

Completed

26-07-2022 16:35

26-07-2022 16:35

00:00:09.9120

2459

170

NA

NA

00:00:00.1180

Cached

NA

The next dataflow takes "Notes" as a linked table, then does some transformations, resulting in a new table called "Clean Notes".  In the query editor, the results of the last step look a bit like this.  Clearly there are multiple rows of data.

After saving that dataflow, I can refresh either only data flow 2 (data cleaning) or dataflows 1 and 2.  In either case, I see the following result for the output of the "Clean Notes" table.  Here only 1 row is processed (and indeed the resulting table has only 1 row).  At the same time, the max commit is 421164, which seems quite large.

Requested on

Dataflow name

Dataflow refresh status

Table name

Partition name

Refresh status

Start time

End time

Duration

Rows processed

Bytes processed (KB)

Max commit (KB)

Processor Time

Wait time

Compute engine

Error

26-07-2022 16:35

Data Cleaning

Completed

Clean Notes

FullRefreshPolicyPartition

Completed

26-07-2022 16:35

26-07-2022 16:35

00:00:19.4270

1

19

421164

00:00:18.8440

00:00:00.0510

Cached + folded

NA

Can anyone help me understand/correct when the query editor shows data but when I refresh the dataflow, the data is not written to the resulting table?  I have experimented with changes to the compute engine, but with no obvious change.  Incremental refresh is disabled for all dataflows. I can provide more information if I've left anything important out of the explanation.  This is my first time implementing linked tables across multiple dataflows, so it could end up being something quite simple, but I've been bashing my head into this wall for several hours now.

[brft]

One additional note--if I disable the compute engine for all three dataflows, the duration increases from about 1 minute to about 8 minutes.  Tolerable in this case, but it gives me great doubt about using the compute engine in the future.  That being said, it does seem to quite clearly be related to some caching of data.  I would understand, I suppose if a key column remained constant and changes in other columns were missed in the refresh, but to miss an update from 1 row to hundreds of rows seems quite odd.  I don't have any keys defined in the flow here--would that help at all?

[jennratten]

Hello - are any of your tables refrencing the disabled table "Clean Details"?

[brft]

No.  Also no issues with this table.  Earlier today I was having a problem where a column was not detected in dataflow 2 that was clearly present in the first dataflow.  Eventually I moved a bit of transformation from dataflow 2 to dataflow one, which I think triggered a change to the schema and emptied the cache.  Not worried exactly about that since it's fixed now, but I suppose it has the same root cause.