Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started

Reply
paulenomial
Frequent Visitor

Error when refreshing dataflow

Hi folks,

 

I've built a dataflow which takes a few tables from a lakehouse in another workspace and loads one of them into a warehouse after some transformation. When I look at the table in the dataflow editor the table will load fine (albeit after a 1m30s delay) but when I try to publish or refresh the dataflow I get this error:

 

inventtrans: Error Code: Mashup Exception Error, Error Details: Couldn't refresh the entity because of an issue with the mashup document MashupException.Error: We're sorry, an error occurred during evaluation. Details: (Request ID: f15df4a5-85a0-43f3-9190-e00ef672f1c4).

 

The dataflow isn't tremendously complex, it just merges queries from a couple tables then adds a few calculated columns in to concatenate some data. Nothing has been done to break query folding from the lakehouse. Does anyone know what this could be?

 

I've tried deleting the query and rebuilding it and I've tried deleting the entire dataflow and rebuilding that as well to no avail. 

8 REPLIES 8
pqian_MSFT
Microsoft Employee
Microsoft Employee

I see that the query didn't fold and it attempted to buffer all the rows locally. 

pqian_MSFT_0-1718910124358.png

 

can you right click on the final entity and select "view query plan", what do you see there?

Hi @pqian_MSFT ,

 

The green indicator on the rows indicates that it is folding however the final step takes approximately 1m 30s to load but the step prior takes only about 20s despite both steps being a concatenation of two fields.

 

See query plan below along with the native data source query:

paulenomial_0-1718954702617.png

select [_].[costamountadjustment] as [costamountadjustment],
    [_].[costamountoperations] as [costamountoperations],
    [_].[costamountphysical] as [costamountphysical],
    [_].[costamountposted] as [costamountposted],
    [_].[dateexpected] as [dateexpected],
    [_].[datefinancial] as [datefinancial],
    [_].[datephysical] as [datephysical],
    [_].[inventdimid] as [inventdimid],
    [_].[inventtransorigin] as [inventtransorigin],
    [_].[invoiceid] as [invoiceid],
    [_].[itemid] as [itemid],
    [_].[packingslipid] as [packingslipid],
    [_].[qty] as [qty],
    [_].[voucher] as [voucher],
    [_].[voucherphysical] as [voucherphysical],
    [_].[dataareaid] as [dataareaid],
    [_].[referenceid] as [referenceid],
    [_].[referencecategory] as [referencecategory],
    [_].[configid] as [configid],
    [_].[inventbatchid] as [inventbatchid],
    [_].[inventcolorid] as [inventcolorid],
    [_].[inventlocationid] as [inventlocationid],
    [_].[inventsiteid] as [inventsiteid],
    [_].[inventsizeid] as [inventsizeid],
    [_].[inventstatusid] as [inventstatusid],
    [_].[inventstyleid] as [inventstyleid],
    [_].[wmslocationid] as [wmslocationid],
    convert(nvarchar(4000), [_].[_FKinventlocation]) as [_FKinventlocation],
    convert(nvarchar(4000), [_].[_FKinventsite]) as [_FKinventsite],
    convert(nvarchar(4000), [_].[_FKinventtable]) as [_FKinventtable]
from 
(
    select [_].[costamountadjustment] as [costamountadjustment],
        [_].[costamountoperations] as [costamountoperations],
        [_].[costamountphysical] as [costamountphysical],
        [_].[costamountposted] as [costamountposted],
        [_].[dateexpected] as [dateexpected],
        [_].[datefinancial] as [datefinancial],
        [_].[datephysical] as [datephysical],
        [_].[inventdimid] as [inventdimid],
        [_].[inventtransorigin] as [inventtransorigin],
        [_].[invoiceid] as [invoiceid],
        [_].[itemid] as [itemid],
        [_].[packingslipid] as [packingslipid],
        [_].[qty] as [qty],
        [_].[voucher] as [voucher],
        [_].[voucherphysical] as [voucherphysical],
        [_].[dataareaid] as [dataareaid],
        [_].[referenceid] as [referenceid],
        [_].[referencecategory] as [referencecategory],
        [_].[configid] as [configid],
        [_].[inventbatchid] as [inventbatchid],
        [_].[inventcolorid] as [inventcolorid],
        [_].[inventlocationid] as [inventlocationid],
        [_].[inventsiteid] as [inventsiteid],
        [_].[inventsizeid] as [inventsizeid],
        [_].[inventstatusid] as [inventstatusid],
        [_].[inventstyleid] as [inventstyleid],
        [_].[wmslocationid] as [wmslocationid],
        [_].[dataareaid] + [_].[inventlocationid] as [_FKinventlocation],
        [_].[dataareaid] + [_].[inventsiteid] as [_FKinventsite],
        [_].[dataareaid] + [_].[itemid] as [_FKinventtable]
    from 
    (
        select [$Outer].[costamountadjustment],
            [$Outer].[costamountoperations],
            [$Outer].[costamountphysical],
            [$Outer].[costamountposted],
            [$Outer].[dateexpected],
            [$Outer].[datefinancial],
            [$Outer].[datephysical],
            [$Outer].[inventdimid],
            [$Outer].[inventtransorigin],
            [$Outer].[invoiceid],
            [$Outer].[itemid],
            [$Outer].[packingslipid],
            [$Outer].[qty],
            [$Outer].[voucher],
            [$Outer].[voucherphysical],
            [$Outer].[dataareaid],
            [$Outer].[referenceid],
            [$Outer].[referencecategory],
            [$Inner].[configid],
            [$Inner].[inventbatchid],
            [$Inner].[inventcolorid],
            [$Inner].[inventlocationid],
            [$Inner].[inventsiteid],
            [$Inner].[inventsizeid],
            [$Inner].[inventstatusid],
            [$Inner].[inventstyleid],
            [$Inner].[wmslocationid]
        from 
        (
            select [$Outer].[costamountadjustment],
                [$Outer].[costamountoperations],
                [$Outer].[costamountphysical],
                [$Outer].[costamountposted],
                [$Outer].[dateexpected],
                [$Outer].[datefinancial],
                [$Outer].[datephysical],
                [$Outer].[inventdimid],
                [$Outer].[inventtransorigin],
                [$Outer].[invoiceid],
                [$Outer].[itemid],
                [$Outer].[packingslipid],
                [$Outer].[qty],
                [$Outer].[voucher],
                [$Outer].[voucherphysical],
                [$Outer].[dataareaid],
                [$Inner].[referenceid],
                [$Inner].[referencecategory]
            from 
            (
                select [costamountadjustment],
                    [costamountoperations],
                    [costamountphysical],
                    [costamountposted],
                    [dateexpected],
                    [datefinancial],
                    [datephysical],
                    [inventdimid],
                    [inventtransorigin],
                    [invoiceid],
                    [itemid],
                    [packingslipid],
                    [qty],
                    [voucher],
                    [voucherphysical],
                    [dataareaid]
                from [dataverse_production_cds2_workspace_unq239aaf6e46184884942eb81311b62].[dbo].[inventtrans] as [$Table]
            ) as [$Outer]
            left outer join 
            (
                select [$Outer].[inventtransid] as [inventtransid],
                    [$Outer].[referenceid] as [referenceid],
                    [$Outer].[dataareaid] as [dataareaid2],
                    [$Outer].[recid] as [recid],
                    [$Inner].[LocalizedLabel] as [referencecategory]
                from 
                (
                    select [referencecategory],
                        [inventtransid],
                        [referenceid],
                        [dataareaid],
                        [recid]
                    from [dataverse_production_cds2_workspace_unq239aaf6e46184884942eb81311b62].[dbo].[inventtransorigin] as [$Table]
                ) as [$Outer]
                left outer join 
                (
                    select [_].[Option],
                        [_].[LocalizedLabel],
                        [_].[ExternalValue]
                    from [dataverse_production_cds2_workspace_unq239aaf6e46184884942eb81311b62].[dbo].[GlobalOptionsetMetadata] as [_]
                    where ([_].[EntityName] = 'inventtransorigin' and [_].[EntityName] is not null) and ([_].[OptionSetName] = 'referencecategory' and [_].[OptionSetName] is not null)
                ) as [$Inner] on ([$Outer].[referencecategory] = [$Inner].[Option] or [$Outer].[referencecategory] is null and [$Inner].[Option] is null)
            ) as [$Inner] on ([$Outer].[inventtransorigin] = [$Inner].[recid] or [$Outer].[inventtransorigin] is null and [$Inner].[recid] is null)
        ) as [$Outer]
        left outer join 
        (
            select [configid] as [configid],
                [inventbatchid] as [inventbatchid],
                [inventcolorid] as [inventcolorid],
                [inventdimid] as [inventdimid2],
                [inventlocationid] as [inventlocationid],
                [inventsiteid] as [inventsiteid],
                [inventsizeid] as [inventsizeid],
                [inventstatusid] as [inventstatusid],
                [inventstyleid] as [inventstyleid],
                [wmslocationid] as [wmslocationid],
                [dataareaid] as [dataareaid2]
            from [dataverse_production_cds2_workspace_unq239aaf6e46184884942eb81311b62].[dbo].[inventdim] as [$Table]
        ) as [$Inner] on (([$Outer].[inventdimid] = [$Inner].[inventdimid2] or [$Outer].[inventdimid] is null and [$Inner].[inventdimid2] is null) and ([$Outer].[dataareaid] = [$Inner].[dataareaid2] or [$Outer].[dataareaid] is null and [$Inner].[dataareaid2] is null))
    ) as [_]
) as [_]

 

This query plan should not be buffering at all.

What is the entity name? The one causing issue is called "inventtrans".

Yes the entity is called inventtrans.

Anonymous
Not applicable

Hi @paulenomial ,

We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet .
In case if you have any resolution please do share that same with the community as it can be helpful to others .

In case if you are still facing the issue, please reach out to our support team so they can do a more thorough investigation on why this it is happening: Link 

 

After creating a Support ticket please provide the ticket number as it would help us to track for more information.

 

Hope this helps. Please let us know if you have any other queries.

Anonymous
Not applicable

Hi @paulenomial ,

We haven’t heard from you on the last response and was just checking back to see if you got a chance to open a support ticket.
After creating a Support ticket please provide the ticket number as it would help us to track for more information.

Thanks

Anonymous
Not applicable

Hi @paulenomial ,

Thanks for using Fabric Community. This might be an intermittent issue, can you please re try it again?
Please let me know incase of further queries.

Hi @Anonymous ,

 

I'm afraid I've tried multiple times over the last few days and it doesn't appear to be going away.

 

Thanks,
Paul.

Helpful resources

Announcements
Sept Fabric Carousel

Fabric Monthly Update - September 2024

Check out the September 2024 Fabric update to learn about new features.

Expanding the Data Factory Forums

New forum boards available in Data Factory

Ask questions in Apache Airflow Job and Mirroring.

September Hackathon Carousel

Microsoft Fabric & AI Learning Hackathon

Learn from experts, get hands-on experience, and win awesome prizes.

Sept NL Carousel

Fabric Community Update - September 2024

Find out what's new and trending in the Fabric Community.