Forum Discussion

JayJay11's avatar
JayJay11
Resolver II
2 years ago

Dataflow runs since hours, cannot be stopped?

Hello,   we are encountering more and more issues with Fabric, unfortunately. The newest addition is a Gen2 Dataflow that runs now since 7 hours while usually it took about 1 hour to complete. I un...
data engineering
pqian_MSFT's avatar
pqian_MSFT
Microsoft Employee
2 years ago

Actually I am not able to see any history on this dataflow indicating that there were 1-hour runs before. Was this perhaps another dataflow? Can you share the 1-hour run dataflow\request IDs?

JayJay11's avatar
JayJay11
Resolver II
2 years ago

Here is the Gen1 Dataflow that runs quite smooth since weeks: a3571cca-34a2-4f0b-8f38-4c53d77378fd

 

And here is the query:

 

let
    Source = OData.Feed("https://myREDACTED-api.s4hana.cloud.sap/sap/opu/odata/sap/API_GLACCOUNTLINEITEM", null, [Implementation="2.0"]),
    GLAccountLineItem_table = Source{[Name="GLAccountLineItem",Signature="table"]}[Data],
    #"Removed Other Columns" = Table.SelectColumns(GLAccountLineItem_table, {"CompanyCode", "AccountingDocument", "Ledger", "GLRecordType", "ControllingArea", "ChartOfAccounts", "GLAccount", "BusinessTransactionType", "CostCenter", "ProfitCenter", "FunctionalArea", "Segment", "PartnerCompany", "GlobalCurrency", "AmountInGlobalCurrency", "BaseUnit", "Quantity", "DebitCreditCode", "FiscalPeriod", "FiscalYearVariant", "FiscalYearPeriod", "PostingDate", "DocumentDate", "AccountingDocumentType", "PostingKey", "LastChangeDateTime", "OriginObjectType", "GLAccountType", "DocumentItemText", "Product", "Supplier", "Customer", "OffsettingAccount", "WBSElementInternalID", "ProjectInternalID"}),
    #"Filter Ledger = 0L" = Table.SelectRows(#"Removed Other Columns", each ([Ledger] = "0L")),
  #"Changed column type" = Table.TransformColumnTypes(#"Filter Ledger = 0L", {{"PostingDate", type date}, {"DocumentDate", type date}, {"LastChangeDateTime", type date}})
in
    #"Changed column type"

 

Again, thank you for checking this. Curious to see the differences between the Gen1 and Gen2 Dataflow ...

  • pqian_MSFT's avatar
    pqian_MSFT
    Microsoft Employee
    2 years ago

    Ah, I had missed the ok-dataflow is Gen1. Do you know if it's running on Premium capacity or shared?

     

    I'll look into the long running time for Gen 2, at a glance it seems to be network bound (specifically, the amount of requests to Odata endpoint). 

     

    • JayJay11's avatar
      JayJay11
      Resolver II
      2 years ago

      It is running on a shared capacity workspace with Premium Per User license.

      • pqian_MSFT's avatar
        pqian_MSFT
        Microsoft Employee
        2 years ago

        I looked into a few runs for your Gen 1 and Gen 2 refresh - the main difference is the time span in OData/GetResponse. There are thousands of these OData requests, they average to 0.5s each on the Gen 1 nodes:

        However, the on the Gen 2 nodes, they average around 3.79s each

        We see the network throughput on Gen 2 node at around 100kbps, and the entire refresh is network IO bound.

        Both the Gen 1 and Gen 2 nodes are from our Germany data center, and so there shouldn't be any latency differences. 

        Is the SAP server perhaps throttling the Gen 2 refresh for whatever reason?