FabCon 2026 Barcelona: Beyond the Announcements
Data Engineers in Toronto October 2026 Semimonthly Meeting Topic: FabCon 2026 Barcelona - Beyond the Announcements Abstract: What was it like to attend Microsoft's flagship Fabric conference in Barcelona? Which announcements generated the most excitement? And more importantly, which ones are likely to matter in real-world customer projects? Join me for an informal recap of FabCon 2026 as I share photos, conference highlights, conversations with Microsoft engineers and community members, and a curated selection of Fabric announcements that caught my attention. Speaker: Alexander Turlov, Principal Consultant at Alta Computing Speaker Profile: Alexander Turlov is a professional consultant, specializing in cloud data architecture and Microsoft Fabric solutions. He helps organizations design scalable data platforms and integrate modern tools like GitHub Copilot and VS Code to accelerate development workflows. Alexander is a certified Microsoft Fabric Data Engineer and an active community contributor, sharing practical insights through presentations and workshops on AI-assisted data engineering and analytics. The meeting is over Microsoft Teams, and the joining link is https://vip.dataengineersintoronto.org/webinar See you at the meeting!
DP-700 exam voucher
Hi Community, I am planning to take DP-700 exam on 20th October 2026. Before I pay for full exam fee, I would like to know if there is any way I can get any voucher(Either 100% or 50% off) to take my exam and while not impacting my wallet:) FYI, I have already prepared so I need to take this week only and I cannot wai for Microsoft Ignite scheduled next month. I'm not qualified for events or levelups intended for Microsoft Partner companie's employee or for Students. Thanks in Advance:)
Architectural Advice: D365 load - Cross Region Fabric link Issue
Hi everyone, We are facing a dilemma due to a shift in our production infrastructure strategy and would love some architectural advice to avoid major rework. Our Current Setup (Dev/UAT): • Dataverse / D365 Environment: Located in Europe. • Fabric Capacity & Workspaces: Located in Europe. • Status: We have already completed the configuration and development in this environment using the native Link to Microsoft Fabric feature. Everything is working perfectly. The Production Challenge: For our Production rollout, leadership wants our main Fabric capacity to reside in the UAE region because Fabric capacity has recently become available there. However, our Production Dataverse environment will remain in Europe. Because the native Link to Microsoft Fabric strictly requires Dataverse and the Fabric capacity to be in the same Azure geography, we cannot link Prod Dataverse directly to a UAE Fabric workspace. Changing our entire integration approach right now would mean throwing away our completed development and starting over. To leverage our current work, we are considering this workaround: 1. Create a Staging Workspace in Europe (matching the Dataverse region) using a small/minimal capacity. 2. Establish the native Link to Microsoft Fabric from the Prod Dataverse environment into this European Staging Workspace. 3. In our main UAE Production Workspace, create OneLake Shortcuts pointing back to the tables synced in the European Staging Workspace. Before we commit to this, we need clarity on a few points: • Is this the best workaround to preserve our current development effort, or does spanning data across Europe and the UAE via OneLake shortcuts introduce severe bottlenecks? • Direct Lake & Query Performance: If we build Power BI reports using Direct Lake mode out of the UAE workspace, will the cross-region network latency (Europe to UAE) degrade report loading times for our end-users? • Licensing Cost Overhead: Does the European staging workspace require a high-tier capacity just to pass data through, or can we run it on a minimal F-SKU (like an F2) just to anchor the Dataverse link? • Alternative Pipelines: If shortcuts across these specific regions are discouraged, should we bite the bullet and rewrite the backend using a Fabric Data Factory Copy Pipeline to physically move data from Europe to the UAE? We would appreciate any insights from anyone who has managed a Europe-to-UAE fabric bridge, or from the Microsoft product team on how to handle this gracefully. Thanks in advance!Git integration and feature branching strategy across Dev Test Prod workspaces in Microsoft Fabric
Hi everyone, We’re setting up a development workflow in Microsoft Fabric with three workspaces: Dev, Test, and Prod. Our items include lakehouses, PySpark notebooks, data pipelines, and semantic models / Power BI reports. We’re using Azure DevOps Git (open to GitHub if it matters). I’d like to understand how teams are handling Git integration and feature development in practice. Git integration across workspaces Do you connect only the Dev workspace to Git, or all three (Dev → dev branch, Test → test, Prod → main)? For promotion to Test and Prod, are you using Fabric Deployment Pipelines, Git-based promotion (PR merges into environment branches), or the fabric-cicd Python library / REST APIs? How do you handle environment-specific values such as lakehouse IDs, connections, and notebook default lakehouse bindings across stages? Feature branches with multiple developers When several features are developed in parallel, how do you set up feature branches? Since a workspace can only be connected to one branch at a time, do you create a separate feature workspace per developer using “Branch out to new workspace”, or does everyone work directly in the shared Dev workspace? If you use feature workspaces, how do you handle lakehouse data? Do notebooks point to the Dev lakehouse, or does each feature workspace get its own lakehouse with sample data? How do you clean up feature workspaces after the PR is merged? Merging and conflicts What’s your PR/merge flow back into the dev branch, and how do you sync the Dev workspace afterward? Have you hit merge conflicts on notebooks, pipelines, or semantic models (TMDL)? How did you resolve them? Gotchas Any item types or limitations that caused problems (unsupported items, shortcuts, SQL endpoint objects, connections)? A short description of your branch structure and promotion flow would help a lot. Thanks in advance!
Procurement analytics from SharePoint, Excel, JSON and on-prem SQL: your Fabric approach?
Hi all, While working on a procurement analytics project at DataToBiz, we ran into a source and reporting problem that I’d like to compare with how people here would handle it in Fabric. The data lived in several places: SharePoint files, Excel and JSON flat files, OneDrive files and a legacy on-prem SQL database. In the original build, we landed everything in Azure Blob storage, converted it to a structured format and fed it into a data warehousing layer, with Power BI on top. The dashboards tracked order management, vendor management and free stock vs. out-of-stock items, along with purchase order counts broken down by plant, month and year. They also covered supplier costs and savings across countries, and the change in raw material costs over time. To keep report loading fast, we moved the workspaces to Premium capacity. That setup predates Fabric, so I’d like to hear how you’d do it today. For the file-based sources, would you use OneLake shortcuts, Dataflows Gen2 or Pipelines? With SharePoint and OneDrive files that change often, what has worked best for you? For the on-prem SQL, would you go through the gateway with a pipeline, or is there a better option now? Where does mirroring fit for a legacy database like that? Would you land everything in a Lakehouse and serve it through a Warehouse, or keep it all in one? We had a structured warehousing layer, and I’m not sure whether the split is worth it now. For the reporting side, would you use Direct Lake or Import mode? Is capacity sizing less of a worry than it was with Premium workspaces, or does it still take effort to get fast report loads? And for things like supplier savings and raw material cost trends, how do you handle currency and country differences in the model? Do you convert in the pipeline or in DAX? Would love to hear how you’d approach it.
Best orchestration choice to reduce Fabric capacity usage when transformations run in Snowflake
Hi, I’m looking for advice on reducing Microsoft Fabric capacity usage. The same Fabric capacity is used for BI, so I want to avoid unnecessary data integration workloads affecting BI. Our current flow is: Fabric Data Factory pipelines ingest data from external sources into ADLS. Fabric Data Factory pipelines load data from ADLS into Snowflake Silver. Fabric Data Factory pipelines trigger Snowflake Silver-to-Gold processing. Most transformations run inside Snowflake. Fabric Data Factory is mainly handling ingestion, loading, and orchestration. These pipelines currently run on Fabric’s native schedules. Would moving orchestration to a Fabric Apache Airflow Job reduce Fabric capacity consumption, or would it add more capacity usage while the Data Factory pipeline activities still run in Fabric? For this type of setup, what approach would you recommend to reduce Fabric capacity impact while keeping reliable scheduling and monitoring?Run OnDemand Fabric Data Pipeline with Runtime Parameters using REST API
Currently, Microsoft Fabric Data Pipelines can be triggered on demand, but there is no native capability to pass runtime parameter values when initiating a pipeline execution through REST APIs. This limitation reduces flexibility for automation, orchestration, and operational support scenarios. FYR - Background Jobs - Run On Demand Execute - REST API (DataPipeline) | Microsoft Learn Problem Statement Many enterprise data integration workloads require dynamic values to be supplied at runtime, such as: Processing a specific business date Running selective backfills Environment-specific executions Customer- or region-specific processing Triggering pipelines from external applications CI/CD and DevOps automation workflows Without runtime parameter support, organizations must create workarounds such as configuration tables, temporary files, or multiple copies of the same pipeline, adding complexity and maintenance overhead. Proposed Enhancement Enhance Fabric Data Pipeline REST APIs to allow: Triggering a pipeline run with runtime parameter values. Overriding default pipeline parameters at execution time. Passing multiple parameter types: String Integer Boolean Decimal Date/DateTime Array/Object Viewing submitted parameter values in monitoring and run history. Invoking pipelines from external applications, Azure Functions, Logic Apps, Power Automate, and DevOps pipelines with custom parameter payloads. Please consider adding support for runtime parameter injection when triggering Fabric Data Pipelines through REST APIs. This capability is critical for enterprise automation and large-scale operational workloads.
Validating a metadata-driven Bronze → Silver incremental load pattern in Fabric
Hi all, I’m building a metadata-driven Bronze → Silver ingestion framework in Microsoft Fabric and would like feedback on the approach before scaling it to a few hundred tables. Setup Source: Dynamics 365 F&O tables land in a Bronze lakehouse as Delta tables through Fabric Link. Each row has a SinkModifiedOn timestamp and an IsDelete soft-delete flag. Config DB: a Fabric SQL Database stores per-table metadata (table name, schema, key column, load type, column list) and the last processed watermark. Orchestration: a Data Pipeline runs a Lookup on the config DB, then a ForEach that calls one generic parameterized notebook per table. The notebook returns the new watermark through notebookutils.notebook.exit(). A Script/Stored Procedure activity writes it back to the config DB only if the notebook succeeds. Target: a Silver lakehouse with one Delta table per source table. Parameter cell (values passed from the pipeline) python table_name = "generaljournalentry" schema = "d365" key_col = "recid" load_type = "incremental" # full / incremental select_cols = "recid,accountingdate,...,SinkModifiedOn,IsDelete" # "" = all columns last_wm = "" # from config DB; "" = first run Load logic python table_name = "generaljournalentry" schema = "d365" key_col = "recid" load_type = "incremental" # full / incremental select_cols = "recid,accountingdate,...,SinkModifiedOn,IsDelete" # "" = all columns last_wm = "" # from config DB; "" = first run df = spark.read.format("delta").load(src_path) if select_cols: df = df.select(*[c.strip() for c in select_cols.split(",")]) new_wm = None # ---------- FULL ---------- if load_type == "full": w = Window.partitionBy(key_col).orderBy(F.col("SinkModifiedOn").desc()) df = (df.withColumn("rn", F.row_number().over(w)).filter("rn = 1").drop("rn") .withColumn("silver_loaded_on", F.current_timestamp())).cache() new_wm = df.agg(F.max("SinkModifiedOn")).first()[0] (df.filter("coalesce(IsDelete, false) = false") .write.format("delta").mode("overwrite").option("overwriteSchema", "true").save(tgt_path)) # ---------- INCREMENTAL ---------- else: target_exists = DeltaTable.isDeltaTable(spark, tgt_path) # Watermark from config DB, with a 1-hour safety overlap if last_wm: df = df.filter(F.col("SinkModifiedOn") > F.to_timestamp(F.lit(last_wm)) - F.expr("INTERVAL 1 HOURS")) # Keep the latest version per key w = Window.partitionBy(key_col).orderBy(F.col("SinkModifiedOn").desc()) df = (df.withColumn("rn", F.row_number().over(w)).filter("rn = 1").drop("rn") .withColumn("silver_loaded_on", F.current_timestamp())).cache() # New watermark = max of the batch (deleted rows included) new_wm = df.agg(F.max("SinkModifiedOn")).first()[0] if new_wm: if not target_exists: df.filter("coalesce(IsDelete, false) = false").write.format("delta").save(tgt_path) else: (DeltaTable.forPath(spark, tgt_path).alias("t") .merge(df.alias("s"), f"t.{key_col} = s.{key_col}") .whenMatchedDelete("coalesce(s.IsDelete, false) = true") .whenMatchedUpdateAll("s.SinkModifiedOn > t.SinkModifiedOn") .whenNotMatchedInsertAll("coalesce(s.IsDelete, false) = false") .execute()) # Return the new watermark to the pipeline notebookutils.notebook.exit(json.dumps({ "table": f"{schema}.{table_name}", "new_watermark": str(new_wm or last_wm) })) Questions Watermark handling. The watermark is passed in from the config DB and written back from the notebook’s exit value only after a successful run, so a failed run never advances it. Is a 1-hour overlap a sensible safety margin for SinkModifiedOn? Is there a cleaner way to keep the watermark update and the Delta MERGE consistent if the pipeline fails between the two steps? SinkModifiedOn reliability. For D365 data landed through Fabric Link, is SinkModifiedOn reliable as a monotonic incremental column? Are there known cases of records arriving with an older timestamp? Deduplication. I keep the latest version per key with row_number() ordered by SinkModifiedOn. Should I add a tiebreaker column, such as versionnumber, to handle identical timestamps? Orchestration at scale. For around 200+ tables, is ForEach → Notebook activity per table reasonable? Or are notebookutils.notebook.runMultiple or high-concurrency sessions the better choice to avoid repeated Spark startup overhead? Soft deletes. Is physically deleting rows in Silver on IsDelete = true the common approach, or do people usually keep soft-deleted rows in Silver and filter them in Gold? Table maintenance. Any recommended settings for frequently merged Silver tables, such as OPTIMIZE or V-Order schedule, deletion vectors, or partitioning? Any feedback, gotchas or alternative patterns would be appreciated. Thanks!
Fabric Copy Job CDC Error On Incremental Load
I'm using the fabric copy job to incrementally copy data from a CDC enabled table on SQL Server 2008 R2. The initial full load ran successfully, however all subsequent runs are failing and returning the following error: Failure happened on 'Source' side. ErrorCode=SqlOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=A database operation failed with the following error: 'Incorrect syntax near '.'.' I'm assuming the error is caused by some dynamic sql script generated by the copy job in the backend. Does anyone know how I can determine what's actually causing this error, and how I can fix it? For context, this error is happening on our production SQL Server 2008. I tested the same copy job on our dev SQL Server 2008 and had no issues whatsoever with the initial load and incremental runs. Both SQL Servers are identical.
