data factory | apache airflow job
20 TopicsData Factory Apache Airflow Job Orchestration
Create seamless Apache Airflow job orchestration in Azure Data Factory to schedule, monitor, and automate pipelines with real-time alerts, retry handling, dependency management, and centralized visibility for reliable data workflows.17Views0likes0CommentsAI-Powered Backend Builder for Microsoft Fabric
Microsoft Fabric could include an AI-powered backend builder that helps developers create secure and scalable application backends with simple commands. It should automatically generate configurations, apply security best practices, and support easy deployment. This would save development time, reduce manual work, and make backend creation easier for both beginners and experienced developers.16Views0likes0CommentsImprove Apache Airflow Job Monitoring in Microsoft Fabric Data Factory
Hello Microsoft Fabric Team, I would like to suggest improving Apache Airflow Job support in Microsoft Fabric Data Factory. Many users run multiple Airflow jobs every day, but it is difficult to monitor them and quickly identify failures. Better monitoring features would improve productivity and reduce troubleshooting time. Suggested improvements: Show the live status of every Airflow job. Display job start time, end time, and total duration. Show clear error messages when a job fails. Allow users to retry failed jobs with one click. Add email or Microsoft Teams notifications for failed jobs. Improve search and filtering options. Display execution logs in a simple and readable format. Show CPU and memory usage for each job. Allow exporting job history to CSV. Provide basic performance recommendations for slow jobs. Benefits: Faster troubleshooting. Better visibility into Airflow jobs. Improved pipeline management. Reduced manual work. Better productivity for data engineers. Better overall user experience. I hope Microsoft Fabric considers adding these improvements in a future update. Thank you for your time and for continuously improving the platform22Views0likes0CommentsAI-Powered Data Insights in Power BI
I would like to suggest adding AI-powered data insights directly within Power BI to help users uncover trends, anomalies, and key takeaways faster. Key benefits: • Auto-generate insights and summaries using AI. • Detect anomalies and outliers in visuals. • Natural language explanations for charts and reports. • Save time and improve data-driven decision making. This feature will make Power BI even more powerful and user-friendly for all users.18Views0likes0CommentsMake employee counts function
Can we make the system work? we can't get employee counts and when we click to submit a report it only loops back to the beginning to pick a report to run. I have no idea what this is under as I am only trying to run reports that management expects every Monday.29Views0likes0CommentsRun Dataflow Gen2 as Service Principal
It's not possible to run a Dataflow Gen2 as Service Principal. This is documented here: Background Jobs - Run On Demand Execute - REST API (Dataflow) | Microsoft Learn It would be great if it was possible to run Dataflow Gen2 as a Service Principal. The ability to run Dataflow Gen2 as Service Principal would enable: identity isolation between environments using dedicated service principals per dev/test/prod environments avoid reliance on user accounts users may leave the project, so we want to avoid reliance on user accounts2.1KViews37likes6CommentsPower BI Apps - Enhance Remove Report Feature from App
There should be a single option to remove reports from App at "Setup" Section itself that should automatically remove audience all together with the deleted report. Currently, it is complex to remove an existing report from Power BI App. First you have to deal with removing Audiences and then remove report from "Content" section. Many a times I get an error and it would not let me remove report from App. Lets try to make this "Delete/Remove Report" form the Power BI App, should be hassel free.346Views0likes0Comments[Bug]: Microsoft Fabric Airflow Jobs Git Sync with Gitlab - Does not support Sub-Groups
ISSUE: When configuring Microsoft Fabric Airflow Jobs with Git Sync in Gitlab, it does not support Repos that belong to Groups and Sub-Groups. Specifically, with PAT: * The following repo "https://gitlab.com/group/test.git" works. * The following repo "https://gitlab.com/group/sub-group/sandbox.git". Cannot be configured!!!! - Althought it shall work, and is a valid Gitlab Repo - The second repo shall be possible to configure, and shall work. This applies to organizations that within a group, are making use of sub-groups. Thanks! KR503Views0likes1CommentAirflow Job API Extension - IP Address
Context: Every Apache Airflow Job get an assigned IP address to their pool. Issue: There is no current way of retrieving the IP address. Without the IP address, it is not possible to dynamically and automatically whitelist the pool to access firewalled resources in Azure. Solution: Provide an API endpoint to retrieve the IP address, and enable it as outputs in Terraform635Views0likes0CommentsAirflow Job Definition Extension - Git Sync
Context: The current job definition in Airflow DOES NOT include details of Git Sync Issue: Replicating environments requires manual configuration of Storage location (i.e. Git) There is now way of automating deployment through API, Terraform or even Workspace git sync. Solution: Add git to the Airflow definition, or an additonal API / Terraform resource619Views0likes0Comments