Forum Discussion

icassiem's avatar
icassiem
Icon for Post Prodigy rankPost Prodigy
2 months ago
Solved

Data Pipeline Maintain & Job Management

Good day,

Following on from this Solved: Data Environment Options Follow-Up - Microsoft Fabric Community

where i ireceived great help and successfully proposed a F2 subscription

 

I am new to Fabric, data factory, python,. here are some questions i need help with please

1. Is data factory like the legacy SSIS but azure version, it remains GUI and second phase i can use more complex python in datafactory like ssis execute c# task or can i split python from GUI datafactory because im selling its code light nad complex pythin is available??

2. so i can do execpetion handling amd email alrets with an smtp to notify failures etc?

3. ther job management like sql managment of steps, fails, alerts etc?

4. When using data factory or python script and new middleware p[atch released does it break coding and depreciate and python/data factory requires maintain/update to datafact/pythoin code to fix?

 

Please Help

Regards

 

 

 

Need Help
  • v-anbandari's avatar
    v-anbandari
    2 months ago

    Hi icassiem ,

    Thank you for reaching out to the Microsoft Fabric  Community.

    Please Consider the below following points :

    1. Yes, Fabric Data Factory is similar to SSIS in that it provides a low-code visual interface for building data pipelines. For more advanced logic, you can use Fabric Notebooks (Python/PySpark) and orchestrate them from your Data Factory pipeline.

    2. Yes, you can implement exception handling using Success/Failure pipeline paths and configure alerts or email notifications for pipeline failures. If you're using Apache Airflow Jobs, SMTP-based email notifications are also supported.

    3. Yes, Fabric provides scheduling, triggers, retries, run history, and monitoring through the Monitoring Hub, serving a similar purpose to SQL Server Agent for pipeline orchestration.

    4. Generally, no. Fabric is a managed service, and routine platform updates typically don't require changes to existing pipelines. However, it's recommended to validate your workloads after major updates and review the release notes, especially when using preview features or external Python libraries.

     For your reference below are the Related Microsoft documentation:

    I hope this helps! Please let us know if you have any further questions.

    Thank you.

4 Replies

  • v-anbandari's avatar
    v-anbandari
    Icon for Community Support rankCommunity Support
    2 months ago

    Hi icassiem ,

    Thank you for reaching out to the Microsoft Fabric  Community.

    Please Consider the below following points :

    1. Yes, Fabric Data Factory is similar to SSIS in that it provides a low-code visual interface for building data pipelines. For more advanced logic, you can use Fabric Notebooks (Python/PySpark) and orchestrate them from your Data Factory pipeline.

    2. Yes, you can implement exception handling using Success/Failure pipeline paths and configure alerts or email notifications for pipeline failures. If you're using Apache Airflow Jobs, SMTP-based email notifications are also supported.

    3. Yes, Fabric provides scheduling, triggers, retries, run history, and monitoring through the Monitoring Hub, serving a similar purpose to SQL Server Agent for pipeline orchestration.

    4. Generally, no. Fabric is a managed service, and routine platform updates typically don't require changes to existing pipelines. However, it's recommended to validate your workloads after major updates and review the release notes, especially when using preview features or external Python libraries.

     For your reference below are the Related Microsoft documentation:

    I hope this helps! Please let us know if you have any further questions.

    Thank you.

    • icassiem's avatar
      icassiem
      Icon for Post Prodigy rankPost Prodigy
      2 months ago

      v-anbandari thank you so much

       

      the issue is more me needing to learn fabric, as i have no knowledge on how to start my F2 envirornment

      I had to voersell just to get little funding, now wondering how, where to start, what is correct, what is best practices and applicable as i need to loop an api per cleint and aggregate pii before insert into silver in the one pipeline process as i have no bronze. so another thread susggested simple pythin pyspark and to gold also python data mainframe sql too gold whs. The pipeline orchestrates and similar to sql agent i can select various tasks of which dataflow and notebooks are one of them i think

       

      then i need to find xamples step by step on how to do this and write the code to ingest the api json pivot and insert to silver lakehouse using pythin pyspark, then another example from silver using pything data mainframe sql to gol

       

      please if you can help with step by step examples and peipeline setup or lakhouse ?7 whs setup

       

      regards 

  • v-anbandari's avatar
    v-anbandari
    Icon for Community Support rankCommunity Support
    2 months ago

    Hi icassiem 
     

    Could you please confirm if your issue has been resolved using the suggested approach? This will help other community members facing similar scenarios.
     

    Thank you.
     

  • v-anbandari's avatar
    v-anbandari
    Icon for Community Support rankCommunity Support
    2 months ago

    Hi icassiem,

    Just checking in to see if your issue has been resolved. If you’re still facing any problems, please feel free to reach out and we’ll be happy to assist further.

    Thank you.