How Can Teams Automate Data Workflows and Reduce Manual Operational Tasks?
As businesses handle increasing volumes of data, managing workflows manually can become time-consuming and difficult to scale. I'm interested in understanding how teams are automating their data operations, particularly when multiple systems and business processes need to work together. A few areas I'd like to explore: How do you automate repetitive data processing and validation tasks? What approaches do you use to handle failed pipeline executions and retry mechanisms? How do you coordinate notifications, approvals, and downstream business processes? Are you using Microsoft Fabric, Power Automate, Azure services, or custom automation solutions? I'd appreciate hearing about practical implementations, tools, and lessons learned from your experience. What has worked best for your team?How Do You Handle Schema Changes in Microsoft Fabric Lakehouses Without Breaking Reports?
Hi everyone, I'm interested in how teams are managing schema evolution in Microsoft Fabric Lakehouses, particularly when source systems frequently introduce new columns, change data types, or rename existing fields. A common challenge in enterprise data engineering is ensuring that changes in upstream systems don't unexpectedly break downstream Delta tables, Spark notebooks, SQL queries, or Power BI semantic models. Microsoft Fabric supports schema evolution for Delta tables, but allowing schema changes automatically isn't always the safest approach for production workloads. I'd be interested in hearing how other data engineers handle a few scenarios: Schema Drift: Do you allow automatic schema evolution using mergeSchema, or do you prefer explicitly approving changes before ingestion? Data Contracts: Are you using formal data contracts between source systems and Fabric Lakehouses to prevent unexpected breaking changes? Pipeline Testing: How do you validate schema compatibility before executing production ingestion and transformation jobs? Downstream Dependencies: What's your approach to identifying affected notebooks, SQL endpoints, and Power BI semantic models before changing a table schema? Recovery Strategies: When a schema change breaks an existing workflow, do you roll back the change, introduce a versioned table, or maintain backward-compatible views? I'm particularly interested in practical solutions that work across Bronze, Silver, and Gold layers in a medallion architecture. One question I'd especially like to explore: Should enterprise data engineering teams prioritize flexible schema evolution or strict data contracts as their Fabric environments scale? Would love to hear what has worked in real production environments, including any lessons learned from unexpected schema changes. Thanks!
Best approach to load report-style Excel files from a SharePoint folder into Lakehouse tables
Hi all, I’d like advice on the best pattern for loading business-maintained Excel files from SharePoint into a Fabric Lakehouse. Scenario Business users upload a monthly Excel file per department to a SharePoint Online document library folder (HSE first; Finance and Internal Audit later). Each file is year-to-date (Jan up to the reported month). The plan is one file per month, named like HSE_Performance_YYYY_MM.xlsx. The sheet is a report layout, not a flat table: two header rows: month names merged across 3 columns, then HQ / Marine / Contractor section header rows inside the data (e.g. “Incident Rates”, “Minor Incidents”) ~120 KPI rows, YTD total columns at the end formula cells that return #DIV/0! The target is about 19 Delta tables (one per KPI section, with KPIs as columns), plus a load log table. Current solution (working in DEV) A PySpark notebook reads the workbook with openpyxl, detects sections, unpivots and pivots, and writes the 19 tables. This works well when the file is uploaded manually to Lakehouse Files. Planned logic: each run, pick the latest file per year; skip files already loaded (name + modified time); replace only that year’s rows; write all tables or none; copy the processed file to an archive folder in Lakehouse Files; log success or failure and alert on failure. Where I need advice OneLake SharePoint shortcut: our service account has “Can view” access directly on the target folder only, not on the parent folders or the library. The shortcut wizard connects at the site URL and browses down. Is folder-level access enough to create the shortcut, or does the identity need access at library or parent-folder level? Has anyone used folder-only permissions successfully? Moving files in SharePoint: as I understand it, SharePoint shortcuts are read-only (writing returns “This operation is not supported through shortcuts of account type OneDriveSharePoint”). Is that correct? Is the Graph API (service principal with Sites.Selected) the recommended way to move processed files to Archive / Error folders in SharePoint? Dataflow Gen2 as a fetch step: Copy activity doesn’t support the SharePoint folder connector, only lists. Has anyone used Dataflow Gen2 just to land the raw sheet into a staging table (all columns as text, errors replaced), then a notebook for the transformation? Any issues with merged cells, error values or column types? Production authentication: for unattended runs, what do you recommend for SharePoint access: workspace identity, a service account (organizational account) or a service principal? Any limitations with workspace identity for SharePoint shortcuts or the SharePoint folder connector? Overall pattern: for monthly, business-maintained, report-style Excel files, would you choose: (a) shortcut + notebook, (b) Dataflow Gen2 fetch + notebook, or (c) notebook calling the Graph API directly? Or is there a better pattern you’ve used? Volumes are small (one file per department per month, under 1 MB), so simplicity and reliability matter more than performance. Thanks in advance!
How are AI agents changing modern data engineering workflows?
I am exploring how AI agents can assist data engineering teams in managing complex data workflows and reducing repetitive operational tasks. Some areas I am interested in: AI agents for monitoring data pipelines and detecting failures Automated data quality checks and anomaly detection Intelligent assistance for ETL/ELT workflow optimization Generating documentation for datasets and transformations Using AI with Fabric pipelines, notebooks, and data workflows With the growth of AI-powered automation, I would like to understand how data engineers are approaching these patterns in real-world environments. What approaches, architectures, or best practices are teams using to combine Microsoft Fabric capabilities with AI agents?
Azure free account - where to see 27$ spent of 200$ free credit and other questions
Hi All, sorry to disturb but I just opened a free account of 1 year 2 days ago and then I am too unexperience using Azure. I tried to install a free database but I used "Central Spain" as server and then failed. I realised it was because "Central Spain" can not be use in free trial. Then I did a new installation using "Central France" and the deploy of the new database went OK. After that I have simplely run very simple queries using the sample database, things like (I am not going to put the exact commands, but it was something like that I write): - select * from table_1 (not even any kind of join) - $myVar = 1 select coalesces($mivVar,2) as coalesce_value and I have things in the next screencapture that I do not understand and that is why I write this post to see if anyone of you could please help me to understand. Question 1: - In orange arrow (right part) it seems I have consumed 28,27$ of my free 200$ credit but If I go to the "Azure subscription 1" details, it says "your remaining $200 of free credit expires in ...".(yellow arrow at the left) I do not understand "remaining" in "your remaining $200" . Do I have 171,73$ (orange arrow) or 200$ as remaining credit? Question 2: Where can I see where I have spent the 28,27$? If you see in my subsription details on the left, it seems I have not consumed any $, so I do not kwow where those 28,27$ were employed, and I doubt that because running around 10 simple queries as the ones I explained I ran it could be that money. Question 3: In left wodn part with purple arrows it seems my subscriptons has 2 databases but if I do go recource manager window, botton right part in screenshot, I only see 1 database. How is this possible? Many thanks to all in advance and sorry for the inconveniences.
