Forum Discussion

ankitmori91's avatar
ankitmori91
Frequent Visitor
1 month ago
Solved

Data pipeline is remaning into InProgress state

Hello,   I have created data pipeline and in that i have only 1 notebook, but it is taking around 15-25 min average to run my pipeline and i am using F4 capacity. and i feel F4 is more then enough ...
data engineering
data pipeline
Spark Job Definition
  • Olufemi7's avatar
    Olufemi7
    1 month ago

    Hello ankitmori91,
    The F4 capacity alone does not tell us why the pipeline is taking 15–25 minutes.

    First check the pipeline run and open the Notebook activity details. Fabric shows the activity duration and performance details in the pipeline run. Check those first to see where the time is being spent.

    Try running the Notebook on its own and see how long it takes compared with the pipeline run.

    If the Notebook itself takes 15–25 minutes, investigate the Notebook execution. If it completes much faster but the pipeline remains InProgress, the pipeline activity details should help narrow down the cause.

    Microsoft Learn: Monitor pipeline runs in Fabric Data Factory

v-achippa's avatar
v-achippa
Icon for Community Support rankCommunity Support
26 days ago

Hi ankitmori91​,

Thank you for reaching out to Microsoft Fabric Community.

Thank you Olufemi7​ and ShivekMaharaj​ for the prompt response.

As we haven’t heard back from you, we wanted to kindly follow up to check if the solution provided by the user's for the issue worked?  or let us know if you need any further assistance.

Thanks and regards,
Anjan Kumar Chippa