Forum Discussion

JayJay11's avatar
JayJay11
Resolver II
2 years ago

Dataflow runs since hours, cannot be stopped?

Hello,   we are encountering more and more issues with Fabric, unfortunately. The newest addition is a Gen2 Dataflow that runs now since 7 hours while usually it took about 1 hour to complete. I un...
data engineering
pqian_MSFT's avatar
pqian_MSFT
Microsoft Employee
2 years ago

JayJay11 can you paste the request ID and/or dataflow ID for this? I want to take a look on why the refresh got stuck

JayJay11's avatar
JayJay11
Resolver II
2 years ago

Here you go:

 

Thank you!!

  • pqian_MSFT's avatar
    pqian_MSFT
    Microsoft Employee
    2 years ago

    Thanks JayJay11  , the difference between the last fast run and this run seems to be on the amount of OData requests. The fast run only issued 4 requests, the slow run is still running and issuing thousands of OData requests.

     

    I'll need to look into these requests to see what they are. Meanwhile, if you can share the query definition, that would be helpful too

    • pqian_MSFT's avatar
      pqian_MSFT
      Microsoft Employee
      2 years ago

      Actually I am not able to see any history on this dataflow indicating that there were 1-hour runs before. Was this perhaps another dataflow? Can you share the 1-hour run dataflow\request IDs?

      • JayJay11's avatar
        JayJay11
        Resolver II
        2 years ago

        Here is the Gen1 Dataflow that runs quite smooth since weeks: a3571cca-34a2-4f0b-8f38-4c53d77378fd

         

        And here is the query:

         

        let
    Source = OData.Feed("https://myREDACTED-api.s4hana.cloud.sap/sap/opu/odata/sap/API_GLACCOUNTLINEITEM", null, [Implementation="2.0"]),
    GLAccountLineItem_table = Source{[Name="GLAccountLineItem",Signature="table"]}[Data],
    #"Removed Other Columns" = Table.SelectColumns(GLAccountLineItem_table, {"CompanyCode", "AccountingDocument", "Ledger", "GLRecordType", "ControllingArea", "ChartOfAccounts", "GLAccount", "BusinessTransactionType", "CostCenter", "ProfitCenter", "FunctionalArea", "Segment", "PartnerCompany", "GlobalCurrency", "AmountInGlobalCurrency", "BaseUnit", "Quantity", "DebitCreditCode", "FiscalPeriod", "FiscalYearVariant", "FiscalYearPeriod", "PostingDate", "DocumentDate", "AccountingDocumentType", "PostingKey", "LastChangeDateTime", "OriginObjectType", "GLAccountType", "DocumentItemText", "Product", "Supplier", "Customer", "OffsettingAccount", "WBSElementInternalID", "ProjectInternalID"}),
    #"Filter Ledger = 0L" = Table.SelectRows(#"Removed Other Columns", each ([Ledger] = "0L")),
  #"Changed column type" = Table.TransformColumnTypes(#"Filter Ledger = 0L", {{"PostingDate", type date}, {"DocumentDate", type date}, {"LastChangeDateTime", type date}})
in
    #"Changed column type"

         

        Again, thank you for checking this. Curious to see the differences between the Gen1 and Gen2 Dataflow ...