DataFlow Gen 2 Error:Fast copy user error

[PolinaOr]

Hi, everyone

Getting the following error when trying to load from Lake House. Other 2 tables from it are working fine. Preview on the errorred table is also fine. I don't have fast copy required for this table.

Error Code: Fast Copy User Error, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: DataSource.Error: Pipeline execution failed (runId: ed5c354c-46f1-47ee-a93f-d31a1ba53837). Operation on target ca-9b399285-c5d3-49ce-9dcd-043b12b9c145 failed: ErrorCode=DeltaSnapshotError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Delta table snapshot doesn't exist.,Source=Microsoft.DataTransfer.ClientLibrary,''Type=Microsoft.Data.DeltaLake.DeltaLakeException,Message=table doesn't exist,Source=Microsoft.Data.DeltaLake,' Details: Reason = DataSource.Error;RunId = ed5c354c-46f1-47ee-a93f-d31a1ba53837 (Request ID: aa7cc8a3-d52e-4a00-bb19-850271030939).

[PolinaOr]

Ok, disabling staging makes it work. Anyone can explain to me why exactly is that the reason foe failed dataflow, when other tables from the source are 100% fine with staging?

[miguel]

hi!

Glad that you were able to find a solution. 

It depends! but would you mind answering the following questions? it'll help us determine what could've happened:

1. What your Dataflow setting for "Fast Copy" was

2. What data it was holding inside your  (did it have any "Binary", "Record" or "Table" values inside the columns?)

[PolinaOr]

Hi,

1) Didnt have Fast Copy enabled for any

2) No, only [Table] values, as far as I understood

[miguel · Accepted solution]

Totally understand that there were no settings at the query level, but would you mind checking if the Fast Copy setting at the Dataflow level was enabled? 

You can see this inside the Options > Dataflow > Scale tab

You can turn this setting off and your Dataflow will not try to use Fast Copy unless you explicitly use the setting at the query level.

Complex values like [Table] values can create some situations that would prevent Fast Copy to behave the way its intended. It also happens with how Staging that table would yield the same column but with null values as you cannot store "Table" (complex values) into a staging table.

[PolinaOr]

Yeah, in the settings the "Allow use of Fast Copy" was allowed. Will turn it off. Should i turn the Staging back on in this case? As I understood, this allows for faster loading of data. Our database is over 10mln lines in each table.

Essentially the Query was creating mismatched results because of this setting?

[miguel]

It's difficult to provide any concrete guidance as I'm not familiar with your scenario or the details of it. We couldn't blame Fast Copy or Staging either.

Fast Copy and staging combined can definitely help you move things quicker into a Dataflow and perhaps utilize an ELT pattern where you can subsequently reference thar query and do your transformations against it.

I'd recommend creating a new thread that is more broad and talks about your scenario in detail so its more about the architectural decision rather than an issue raised by Fast Copy. Even suggesting something on just the decision of Fast Copy and Staging would be too limiting or too scoped and would definitely recommend looking at the whole scenario in detail so the most optimal architectural decision can be reached.

[PolinaOr]

Thank you very much!