Forum Discussion

JayJay11's avatar
JayJay11
Resolver II
2 years ago

Dataflow runs since hours, cannot be stopped?

Hello,   we are encountering more and more issues with Fabric, unfortunately. The newest addition is a Gen2 Dataflow that runs now since 7 hours while usually it took about 1 hour to complete. I un...
data engineering
JayJay11's avatar
JayJay11
Resolver II
2 years ago

Here you go:

 

Thank you!!

pqian_MSFT's avatar
pqian_MSFT
Microsoft Employee
2 years ago

Thanks JayJay11  , the difference between the last fast run and this run seems to be on the amount of OData requests. The fast run only issued 4 requests, the slow run is still running and issuing thousands of OData requests.

 

I'll need to look into these requests to see what they are. Meanwhile, if you can share the query definition, that would be helpful too

  • pqian_MSFT's avatar
    pqian_MSFT
    Microsoft Employee
    2 years ago

    Actually I am not able to see any history on this dataflow indicating that there were 1-hour runs before. Was this perhaps another dataflow? Can you share the 1-hour run dataflow\request IDs?

    • JayJay11's avatar
      JayJay11
      Resolver II
      2 years ago

      Here is the Gen1 Dataflow that runs quite smooth since weeks: a3571cca-34a2-4f0b-8f38-4c53d77378fd

       

      And here is the query:

       

      let
    Source = OData.Feed("https://myREDACTED-api.s4hana.cloud.sap/sap/opu/odata/sap/API_GLACCOUNTLINEITEM", null, [Implementation="2.0"]),
    GLAccountLineItem_table = Source{[Name="GLAccountLineItem",Signature="table"]}[Data],
    #"Removed Other Columns" = Table.SelectColumns(GLAccountLineItem_table, {"CompanyCode", "AccountingDocument", "Ledger", "GLRecordType", "ControllingArea", "ChartOfAccounts", "GLAccount", "BusinessTransactionType", "CostCenter", "ProfitCenter", "FunctionalArea", "Segment", "PartnerCompany", "GlobalCurrency", "AmountInGlobalCurrency", "BaseUnit", "Quantity", "DebitCreditCode", "FiscalPeriod", "FiscalYearVariant", "FiscalYearPeriod", "PostingDate", "DocumentDate", "AccountingDocumentType", "PostingKey", "LastChangeDateTime", "OriginObjectType", "GLAccountType", "DocumentItemText", "Product", "Supplier", "Customer", "OffsettingAccount", "WBSElementInternalID", "ProjectInternalID"}),
    #"Filter Ledger = 0L" = Table.SelectRows(#"Removed Other Columns", each ([Ledger] = "0L")),
  #"Changed column type" = Table.TransformColumnTypes(#"Filter Ledger = 0L", {{"PostingDate", type date}, {"DocumentDate", type date}, {"LastChangeDateTime", type date}})
in
    #"Changed column type"

       

      Again, thank you for checking this. Curious to see the differences between the Gen1 and Gen2 Dataflow ...

      • pqian_MSFT's avatar
        pqian_MSFT
        Microsoft Employee
        2 years ago

        Ah, I had missed the ok-dataflow is Gen1. Do you know if it's running on Premium capacity or shared?

         

        I'll look into the long running time for Gen 2, at a glance it seems to be network bound (specifically, the amount of requests to Odata endpoint).