Error when refreshing dataflow

[paulenomial]

Hi folks,

I've built a dataflow which takes a few tables from a lakehouse in another workspace and loads one of them into a warehouse after some transformation. When I look at the table in the dataflow editor the table will load fine (albeit after a 1m30s delay) but when I try to publish or refresh the dataflow I get this error:

inventtrans: Error Code: Mashup Exception Error, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: We're sorry, an error occurred during evaluation. Details: (Request ID: f15df4a5-85a0-43f3-9190-e00ef672f1c4).

The dataflow isn't tremendously complex, it just merges queries from a couple tables then adds a few calculated columns in to concatenate some data. Nothing has been done to break query folding from the lakehouse. Does anyone know what this could be?

I've tried deleting the query and rebuilding it and I've tried deleting the entire dataflow and rebuilding that as well to no avail. 

[pqian_MSFT]

I see that the query didn't fold and it attempted to buffer all the rows locally. 

can you right click on the final entity and select "view query plan", what do you see there?

[paulenomial]

Hi @pqian_MSFT ,

The green indicator on the rows indicates that it is folding however the final step takes approximately 1m 30s to load but the step prior takes only about 20s despite both steps being a concatenation of two fields.

See query plan below along with the native data source query:

select [_].[costamountadjustment] as [costamountadjustment],
    [_].[costamountoperations] as [costamountoperations],
    [_].[costamountphysical] as [costamountphysical],
    [_].[costamountposted] as [costamountposted],
    [_].[dateexpected] as [dateexpected],
    [_].[datefinancial] as [datefinancial],
    [_].[datephysical] as [datephysical],
    [_].[inventdimid] as [inventdimid],
    [_].[inventtransorigin] as [inventtransorigin],
    [_].[invoiceid] as [invoiceid],
    [_].[itemid] as [itemid],
    [_].[packingslipid] as [packingslipid],
    [_].[qty] as [qty],
    [_].[voucher] as [voucher],
    [_].[voucherphysical] as [voucherphysical],
    [_].[dataareaid] as [dataareaid],
    [_].[referenceid] as [referenceid],
    [_].[referencecategory] as [referencecategory],
    [_].[configid] as [configid],
    [_].[inventbatchid] as [inventbatchid],
    [_].[inventcolorid] as [inventcolorid],
    [_].[inventlocationid] as [inventlocationid],
    [_].[inventsiteid] as [inventsiteid],
    [_].[inventsizeid] as [inventsizeid],
    [_].[inventstatusid] as [inventstatusid],
    [_].[inventstyleid] as [inventstyleid],
    [_].[wmslocationid] as [wmslocationid],
    convert(nvarchar(4000), [_].[_FKinventlocation]) as [_FKinventlocation],
    convert(nvarchar(4000), [_].[_FKinventsite]) as [_FKinventsite],
    convert(nvarchar(4000), [_].[_FKinventtable]) as [_FKinventtable]
from 
(
    select [_].[costamountadjustment] as [costamountadjustment],
        [_].[costamountoperations] as [costamountoperations],
        [_].[costamountphysical] as [costamountphysical],
        [_].[costamountposted] as [costamountposted],
        [_].[dateexpected] as [dateexpected],
        [_].[datefinancial] as [datefinancial],
        [_].[datephysical] as [datephysical],
        [_].[inventdimid] as [inventdimid],
        [_].[inventtransorigin] as [inventtransorigin],
        [_].[invoiceid] as [invoiceid],
        [_].[itemid] as [itemid],
        [_].[packingslipid] as [packingslipid],
        [_].[qty] as [qty],
        [_].[voucher] as [voucher],
        [_].[voucherphysical] as [voucherphysical],
        [_].[dataareaid] as [dataareaid],
        [_].[referenceid] as [referenceid],
        [_].[referencecategory] as [referencecategory],
        [_].[configid] as [configid],
        [_].[inventbatchid] as [inventbatchid],
        [_].[inventcolorid] as [inventcolorid],
        [_].[inventlocationid] as [inventlocationid],
        [_].[inventsiteid] as [inventsiteid],
        [_].[inventsizeid] as [inventsizeid],
        [_].[inventstatusid] as [inventstatusid],
        [_].[inventstyleid] as [inventstyleid],
        [_].[wmslocationid] as [wmslocationid],
        [_].[dataareaid] + [_].[inventlocationid] as [_FKinventlocation],
        [_].[dataareaid] + [_].[inventsiteid] as [_FKinventsite],
        [_].[dataareaid] + [_].[itemid] as [_FKinventtable]
    from 
    (
        select [$Outer].[costamountadjustment],
            [$Outer].[costamountoperations],
            [$Outer].[costamountphysical],
            [$Outer].[costamountposted],
            [$Outer].[dateexpected],
            [$Outer].[datefinancial],
            [$Outer].[datephysical],
            [$Outer].[inventdimid],
            [$Outer].[inventtransorigin],
            [$Outer].[invoiceid],
            [$Outer].[itemid],
            [$Outer].[packingslipid],
            [$Outer].[qty],
            [$Outer].[voucher],
            [$Outer].[voucherphysical],
            [$Outer].[dataareaid],
            [$Outer].[referenceid],
            [$Outer].[referencecategory],
            [$Inner].[configid],
            [$Inner].[inventbatchid],
            [$Inner].[inventcolorid],
            [$Inner].[inventlocationid],
            [$Inner].[inventsiteid],
            [$Inner].[inventsizeid],
            [$Inner].[inventstatusid],
            [$Inner].[inventstyleid],
            [$Inner].[wmslocationid]
        from 
        (
            select [$Outer].[costamountadjustment],
                [$Outer].[costamountoperations],
                [$Outer].[costamountphysical],
                [$Outer].[costamountposted],
                [$Outer].[dateexpected],
                [$Outer].[datefinancial],
                [$Outer].[datephysical],
                [$Outer].[inventdimid],
                [$Outer].[inventtransorigin],
                [$Outer].[invoiceid],
                [$Outer].[itemid],
                [$Outer].[packingslipid],
                [$Outer].[qty],
                [$Outer].[voucher],
                [$Outer].[voucherphysical],
                [$Outer].[dataareaid],
                [$Inner].[referenceid],
                [$Inner].[referencecategory]
            from 
            (
                select [costamountadjustment],
                    [costamountoperations],
                    [costamountphysical],
                    [costamountposted],
                    [dateexpected],
                    [datefinancial],
                    [datephysical],
                    [inventdimid],
                    [inventtransorigin],
                    [invoiceid],
                    [itemid],
                    [packingslipid],
                    [qty],
                    [voucher],
                    [voucherphysical],
                    [dataareaid]
                from [dataverse_production_cds2_workspace_unq239aaf6e46184884942eb81311b62].[dbo].[inventtrans] as [$Table]
            ) as [$Outer]
            left outer join 
            (
                select [$Outer].[inventtransid] as [inventtransid],
                    [$Outer].[referenceid] as [referenceid],
                    [$Outer].[dataareaid] as [dataareaid2],
                    [$Outer].[recid] as [recid],
                    [$Inner].[LocalizedLabel] as [referencecategory]
                from 
                (
                    select [referencecategory],
                        [inventtransid],
                        [referenceid],
                        [dataareaid],
                        [recid]
                    from [dataverse_production_cds2_workspace_unq239aaf6e46184884942eb81311b62].[dbo].[inventtransorigin] as [$Table]
                ) as [$Outer]
                left outer join 
                (
                    select [_].[Option],
                        [_].[LocalizedLabel],
                        [_].[ExternalValue]
                    from [dataverse_production_cds2_workspace_unq239aaf6e46184884942eb81311b62].[dbo].[GlobalOptionsetMetadata] as [_]
                    where ([_].[EntityName] = 'inventtransorigin' and [_].[EntityName] is not null) and ([_].[OptionSetName] = 'referencecategory' and [_].[OptionSetName] is not null)
                ) as [$Inner] on ([$Outer].[referencecategory] = [$Inner].[Option] or [$Outer].[referencecategory] is null and [$Inner].[Option] is null)
            ) as [$Inner] on ([$Outer].[inventtransorigin] = [$Inner].[recid] or [$Outer].[inventtransorigin] is null and [$Inner].[recid] is null)
        ) as [$Outer]
        left outer join 
        (
            select [configid] as [configid],
                [inventbatchid] as [inventbatchid],
                [inventcolorid] as [inventcolorid],
                [inventdimid] as [inventdimid2],
                [inventlocationid] as [inventlocationid],
                [inventsiteid] as [inventsiteid],
                [inventsizeid] as [inventsizeid],
                [inventstatusid] as [inventstatusid],
                [inventstyleid] as [inventstyleid],
                [wmslocationid] as [wmslocationid],
                [dataareaid] as [dataareaid2]
            from [dataverse_production_cds2_workspace_unq239aaf6e46184884942eb81311b62].[dbo].[inventdim] as [$Table]
        ) as [$Inner] on (([$Outer].[inventdimid] = [$Inner].[inventdimid2] or [$Outer].[inventdimid] is null and [$Inner].[inventdimid2] is null) and ([$Outer].[dataareaid] = [$Inner].[dataareaid2] or [$Outer].[dataareaid] is null and [$Inner].[dataareaid2] is null))
    ) as [_]
) as [_]

[pqian_MSFT]

This query plan should not be buffering at all.

What is the entity name? The one causing issue is called "inventtrans".

[paulenomial]

Yes the entity is called inventtrans.

[Anonymous]

Hi @paulenomial ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .

In case if you are still facing the issue, please reach out to our support team so they can do a more thorough investigation on why this it is happening: Link 

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Hope this helps. Please let us know if you have any other queries.

[Anonymous]

Hi @paulenomial ,

We haven’t heard from you on the last response and was just checking back to see if you got a chance to open a support ticket.
After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thanks

[Anonymous]

Hi @paulenomial ,

Thanks for using Fabric Community. This might be an intermittent issue, can you please re try it again?
Please let me know incase of further queries.

[paulenomial]

Hi @Anonymous ,

I'm afraid I've tried multiple times over the last few days and it doesn't appear to be going away.

Thanks,
Paul.