Forum Discussion
paulenomial
2 years agoFrequent Visitor
Error when refreshing dataflow
Hi folks, I've built a dataflow which takes a few tables from a lakehouse in another workspace and loads one of them into a warehouse after some transformation. When I look at the table in the da...
pqian_MSFT
2 years agoMicrosoft Employee
I see that the query didn't fold and it attempted to buffer all the rows locally.
can you right click on the final entity and select "view query plan", what do you see there?
- paulenomial2 years agoFrequent Visitor
Hi pqian_MSFT ,
The green indicator on the rows indicates that it is folding however the final step takes approximately 1m 30s to load but the step prior takes only about 20s despite both steps being a concatenation of two fields.
See query plan below along with the native data source query:
select [_].[costamountadjustment] as [costamountadjustment], [_].[costamountoperations] as [costamountoperations], [_].[costamountphysical] as [costamountphysical], [_].[costamountposted] as [costamountposted], [_].[dateexpected] as [dateexpected], [_].[datefinancial] as [datefinancial], [_].[datephysical] as [datephysical], [_].[inventdimid] as [inventdimid], [_].[inventtransorigin] as [inventtransorigin], [_].[invoiceid] as [invoiceid], [_].[itemid] as [itemid], [_].[packingslipid] as [packingslipid], [_].[qty] as [qty], [_].[voucher] as [voucher], [_].[voucherphysical] as [voucherphysical], [_].[dataareaid] as [dataareaid], [_].[referenceid] as [referenceid], [_].[referencecategory] as [referencecategory], [_].[configid] as [configid], [_].[inventbatchid] as [inventbatchid], [_].[inventcolorid] as [inventcolorid], [_].[inventlocationid] as [inventlocationid], [_].[inventsiteid] as [inventsiteid], [_].[inventsizeid] as [inventsizeid], [_].[inventstatusid] as [inventstatusid], [_].[inventstyleid] as [inventstyleid], [_].[wmslocationid] as [wmslocationid], convert(nvarchar(4000), [_].[_FKinventlocation]) as [_FKinventlocation], convert(nvarchar(4000), [_].[_FKinventsite]) as [_FKinventsite], convert(nvarchar(4000), [_].[_FKinventtable]) as [_FKinventtable] from ( select [_].[costamountadjustment] as [costamountadjustment], [_].[costamountoperations] as [costamountoperations], [_].[costamountphysical] as [costamountphysical], [_].[costamountposted] as [costamountposted], [_].[dateexpected] as [dateexpected], [_].[datefinancial] as [datefinancial], [_].[datephysical] as [datephysical], [_].[inventdimid] as [inventdimid], [_].[inventtransorigin] as [inventtransorigin], [_].[invoiceid] as [invoiceid], [_].[itemid] as [itemid], [_].[packingslipid] as [packingslipid], [_].[qty] as [qty], [_].[voucher] as [voucher], [_].[voucherphysical] as [voucherphysical], [_].[dataareaid] as [dataareaid], [_].[referenceid] as [referenceid], [_].[referencecategory] as [referencecategory], [_].[configid] as [configid], [_].[inventbatchid] as [inventbatchid], [_].[inventcolorid] as [inventcolorid], [_].[inventlocationid] as [inventlocationid], [_].[inventsiteid] as [inventsiteid], [_].[inventsizeid] as [inventsizeid], [_].[inventstatusid] as [inventstatusid], [_].[inventstyleid] as [inventstyleid], [_].[wmslocationid] as [wmslocationid], [_].[dataareaid] + [_].[inventlocationid] as [_FKinventlocation], [_].[dataareaid] + [_].[inventsiteid] as [_FKinventsite], [_].[dataareaid] + [_].[itemid] as [_FKinventtable] from ( select [$Outer].[costamountadjustment], [$Outer].[costamountoperations], [$Outer].[costamountphysical], [$Outer].[costamountposted], [$Outer].[dateexpected], [$Outer].[datefinancial], [$Outer].[datephysical], [$Outer].[inventdimid], [$Outer].[inventtransorigin], [$Outer].[invoiceid], [$Outer].[itemid], [$Outer].[packingslipid], [$Outer].[qty], [$Outer].[voucher], [$Outer].[voucherphysical], [$Outer].[dataareaid], [$Outer].[referenceid], [$Outer].[referencecategory], [$Inner].[configid], [$Inner].[inventbatchid], [$Inner].[inventcolorid], [$Inner].[inventlocationid], [$Inner].[inventsiteid], [$Inner].[inventsizeid], [$Inner].[inventstatusid], [$Inner].[inventstyleid], [$Inner].[wmslocationid] from ( select [$Outer].[costamountadjustment], [$Outer].[costamountoperations], [$Outer].[costamountphysical], [$Outer].[costamountposted], [$Outer].[dateexpected], [$Outer].[datefinancial], [$Outer].[datephysical], [$Outer].[inventdimid], [$Outer].[inventtransorigin], [$Outer].[invoiceid], [$Outer].[itemid], [$Outer].[packingslipid], [$Outer].[qty], [$Outer].[voucher], [$Outer].[voucherphysical], [$Outer].[dataareaid], [$Inner].[referenceid], [$Inner].[referencecategory] from ( select [costamountadjustment], [costamountoperations], [costamountphysical], [costamountposted], [dateexpected], [datefinancial], [datephysical], [inventdimid], [inventtransorigin], [invoiceid], [itemid], [packingslipid], [qty], [voucher], [voucherphysical], [dataareaid] from [dataverse_production_cds2_workspace_unq239aaf6e46184884942eb81311b62].[dbo].[inventtrans] as [$Table] ) as [$Outer] left outer join ( select [$Outer].[inventtransid] as [inventtransid], [$Outer].[referenceid] as [referenceid], [$Outer].[dataareaid] as [dataareaid2], [$Outer].[recid] as [recid], [$Inner].[LocalizedLabel] as [referencecategory] from ( select [referencecategory], [inventtransid], [referenceid], [dataareaid], [recid] from [dataverse_production_cds2_workspace_unq239aaf6e46184884942eb81311b62].[dbo].[inventtransorigin] as [$Table] ) as [$Outer] left outer join ( select [_].[Option], [_].[LocalizedLabel], [_].[ExternalValue] from [dataverse_production_cds2_workspace_unq239aaf6e46184884942eb81311b62].[dbo].[GlobalOptionsetMetadata] as [_] where ([_].[EntityName] = 'inventtransorigin' and [_].[EntityName] is not null) and ([_].[OptionSetName] = 'referencecategory' and [_].[OptionSetName] is not null) ) as [$Inner] on ([$Outer].[referencecategory] = [$Inner].[Option] or [$Outer].[referencecategory] is null and [$Inner].[Option] is null) ) as [$Inner] on ([$Outer].[inventtransorigin] = [$Inner].[recid] or [$Outer].[inventtransorigin] is null and [$Inner].[recid] is null) ) as [$Outer] left outer join ( select [configid] as [configid], [inventbatchid] as [inventbatchid], [inventcolorid] as [inventcolorid], [inventdimid] as [inventdimid2], [inventlocationid] as [inventlocationid], [inventsiteid] as [inventsiteid], [inventsizeid] as [inventsizeid], [inventstatusid] as [inventstatusid], [inventstyleid] as [inventstyleid], [wmslocationid] as [wmslocationid], [dataareaid] as [dataareaid2] from [dataverse_production_cds2_workspace_unq239aaf6e46184884942eb81311b62].[dbo].[inventdim] as [$Table] ) as [$Inner] on (([$Outer].[inventdimid] = [$Inner].[inventdimid2] or [$Outer].[inventdimid] is null and [$Inner].[inventdimid2] is null) and ([$Outer].[dataareaid] = [$Inner].[dataareaid2] or [$Outer].[dataareaid] is null and [$Inner].[dataareaid2] is null)) ) as [_] ) as [_]- pqian_MSFT2 years agoMicrosoft Employee
This query plan should not be buffering at all.
What is the entity name? The one causing issue is called "inventtrans".
- paulenomial2 years agoFrequent Visitor
Yes the entity is called inventtrans.