How SSIS will work on Fabric
Hello We are trying to migrate to fabric from Synapse. We have ETL currently in synapse. and Azure sql db as the DWH. Apart from that we have two azure apps which use SSIS (and created SSISDB) in Azure sql db We use SSIS IR in Synapse. The Azure apps are build on Angular code. One app uploads files and which lands in Data lake gen 2 now and then we process this data into Azure sql db. How it would work in Migration SSIS into Fabric. I know about that we can land packages in LakeHouse and use Fabric Pipelines to invoke SSIS packages. Please can someone gives me some thorough information how this will work and SSIS from Sysnapse can work on Fabric? Thanks in advance!
Shared data, one consistent view across teams.
Healthcare analytics is a good example of why this matters. Clinical teams, operations teams, and management may need different reports, but those reports still depend on reliable, consistent patient data. If information sits across operational systems and SharePoint files, preparing it for reporting can become a challenge of integration and data quality before visualization even begins. One detail I found interesting in a Microsoft Fabric healthcare analytics project we recently took for datatobiz is the use of OneLake alongside automated data processing and Power BI dashboards. The architectural point is worth discussing: bringing data into a common environment is only the first step. The next is making sure the data is appropriately prepared, governed, and structured for the people using it. For those working with Fabric, how are you organizing data from multiple sources before it reaches the reporting layer? Do you prefer a Lakehouse-first approach, a Warehouse-first approach, or a combination based on the workload?
Completed DP-600 Practice Assessment (Score: 90%) - Tracking for 100% Voucher Selection
Hi everyone, I have just completed the official DP-600 practice assessment and achieved a score of 90%. I applied for the 100% certification voucher program, and I am posting this to ensure my completion is visible and recorded in case I am selected. Currently, my assessment score and completion status are not showing up under the Credentials tab on my Microsoft Learn profile. Since I want to make sure my eligibility is fully tracked for the voucher issuance, I wanted to raise this here. Has anyone else noticed a delay in their practice test scores syncing to their profile? Is there any additional step required to ensure the Fabric community/review team can verify my 90% score? Thank you for your help!
Fabric APPs built on Lakehouse deployment error: 400 source attachment failed
Hello i had tried to create a Fabric app built on a lakehouse connector as described here: https://learn.microsoft.com/en-us/fabric/apps/connectors and i had followed the following steps: 1-Added a fabric-sqlanalytics connector with read-only operations using the Rayfin CLI. The configured source is the Lakehouse SQL analytics endpoint. 2-Confirmed connector metadata was discovered. A CLI inspection of the source returned table metadata and a sample for the original table. 3-Installed the CLI-printed connector dependencies at version 1.36.2. 4-Defined a read-only entity for DEAL_Table, mapping only DEAL_ID as nullable text. No primary key was declared because the discovered metadata marks DEAL_ID nullable. 5-Registered the entity in the connector schema and frontend client. The app query selects dealId and limits the request to one row. 6-Ran local validation; typecheck and focused app tests passed. 7-Tried deployment with the default application authentication. Connector attachment failed with the same 400 error, although the overall deployment continued and published the frontend. 8-Changed the connector to delegated authentication and ran a full rayfin up --json. That run returned overall success, but connector generation reported a local unlink error for .temp/rayfin.tsbuildinfo, so it did not confirm connector application. 9-Retried with rayfin up connector apply --name .... Generation succeeded, but the server again returned 400 Source attachment failed, with the latest activity ID above. The browser app cannot successfully query this connector because it has not attached. any help is more than welcome?
Microsoft Fabric SAP Table Copy Job Fails During Incremental Load with `__rowMarker__` Column Error
Hey Everyone, I am currently working on a Microsoft Fabric Copy Job, using SAP Table Application Server as the source to fetch data from SAP HANA. I have enabled the ABAP Add-on option and configured a Fabric SQL Database as the destination, with the copy mode set to Incremental Load. The initial full load completes successfully, and the data is loaded into the destination table. However, when I modify an existing document in SAP and rerun the Copy Job, it fails with the following error: Error: SqlColumnNameNotExist: Column '__rowMarker__' does not exist in the target table '[##InterimTable_db7db620-2727-4b15-8095-e5aab4f179f8]'. Has anyone encountered this issue with Microsoft Fabric SAP Table Copy Jobs while using incremental loading with the ABAP Add-on? Any suggestions on how to resolve this issue would be appreciated. I have attached a screenshot of the error for reference. Any help in identifying the root cause and resolving this issue would be appreciated. Regards, Darshan
FabricPipelineWorker.exe crash with SAP connection
Hello all, We are experiencing recurring crashes of FabricPipelineWorker.exe when using Microsoft Fabric Data pipelines to extract data from SAP through the SAP Table Application Server connector and an On-premises Data Gateway. The issue appears to be similar to the one reported in this discussion: https://community.fabric.microsoft.com/discussions/df_pipelines/faulting-application-fabricpipelineworker-exe/5364506 For some reason I cannot reply on mentioned post with message: "This content was not accepted by moderation." However, despite updating the relevant components and changing the gateway configuration, the issue persists. Issue description The interesting part is that Copy activities complete successfully in Microsoft Fabric, and we do not see any errors or failed pipeline executions in Fabric itself. In the On-premises Data Gateway logs, the Copy activities also return: Status: Succeeded However, FabricPipelineWorker.exe appears to crash after the data transfer has completed, potentially during the cleanup phase when existing SAP connections are being closed or disposed of. This is currently our assumption based on the observed behavior, rather than a confirmed root cause. Although the operations complete successfully, these crashes generate a very large number of error logs on the SAP side, creating additional operational overhead and making monitoring more difficult. Environment On-premises Data Gateway: 3000.334.4 Microsoft C++ Runtime 14.x: 14.50.35719.0 SAP .NET Connector (NCo): 3.1.8.0 Troubleshooting performed We have already taken the following steps: Updated the On-premises Data Gateway to version 3000.334.4. Updated the SAP .NET Connector to version 3.1.8.0. Verified the Microsoft C++ Runtime version. Disabled gateway load balancing to rule out potential issues with connections being distributed across gateway nodes. Reviewed the gateway logs and confirmed that Copy activities report successful completion. Unfortunately, none of these changes has resolved the issue. Since the data transfers themselves are successful, this appears to be more related to the gateway worker process lifecycle or SAP connection cleanup than to SAP connectivity or data extraction. Any insights from the Microsoft Fabric engineering team or others experiencing similar behavior would be greatly appreciated. Thank you!Copilot Studio, Foundry, Fabric Data Agent or Operations Agent? Which one to pick
Most confusion comes from the word "agent" being used for all four. The fastest way to choose is to ask two questions: who builds it (maker or developer) and what triggers it (a person asking, or data changing). Meet the four Copilot Studio is Microsoft's low-code agent builder on Power Platform. You describe the agent in plain language, add knowledge (SharePoint, Dataverse, websites), plug in 1,000+ connectors and Power Automate flows, then publish to Teams, Microsoft 365 Copilot or a website. It is also the easiest place to orchestrate other agents, including Foundry and Fabric agents. Microsoft Foundry (formerly Azure AI Foundry) is the pro-code platform for engineers. You pick from a large model catalog, write orchestration in code, run evaluations, trace every call and deploy behind private networking. Use it when the agent is a product component, not just a chat window. Fabric data agent is a read-only, conversational analyst over your Fabric data. It turns a question into SQL, DAX or KQL against up to five sources (lakehouse, warehouse, Power BI semantic model, KQL database, ontology) and respects each user's permissions. It is generally available. Fabric operations agent lives in Real-Time Intelligence. You give it an Eventhouse, a business goal and allowed actions. It writes its own monitoring rules, wakes up when a condition hits, reasons over the data, and messages people in Teams or runs a Power Automate action. It reached general availability in mid-2026. Which one should I pick? Ask the questions in order and stop at the first "yes". A "yes" does not rule the others out: a data agent or Foundry agent is often published through Copilot Studio. Side by side Copilot Studio Foundry Fabric data agent Fabric operations agent Core job Converse and automate Build custom AI apps and agents Answer questions over data Monitor live data and act Triggered by A person (or event/flow) Your code A person or another agent A data condition Builder Maker, IT, business AI engineer, developer Data/BI team Ops and RTI team Skill level Low-code Pro-code Low-code + good data modelling Low-code, chat-based setup Data it reaches Connectors, SharePoint, Dataverse, other agents Anything you wire up Lakehouse, warehouse, semantic model, KQL, ontology Eventhouse (KQL) streams Writes or acts? Yes, via flows and tools Yes, whatever you code No, read-only Yes, Teams messages and Power Automate actions Where users meet it Teams, M365 Copilot, web, other channels Your app, API, Teams Fabric, Copilot Studio, Foundry, M365 Copilot, MCP Teams chat or channel Billing Copilot Credits Azure consumption Fabric capacity (CU) Fabric capacity (CU) Pick it when You need speed and M365 reach You need model choice, evals, isolation People Real-world scenarios These are the patterns that come up most often in the community. Each names the tool that fits and why. HR policy helper in Teams → Copilot Studio. A 3,000-person company wants staff to ask "How many leave days do I have?" and "Book Thursday off." Knowledge sits in SharePoint, leave balances in an HR system with a connector. A maker builds it in days and publishes to Teams. No developer needed. Insurance claims triage inside a customer app → Foundry. An insurer reads uploaded photos and PDFs, extracts fields, scores fraud risk and returns a decision to its claims system through an API. It needs a vision model, evaluation datasets, tracing and a private network. That is a product feature, so it belongs in Foundry. Sales leaders asking about pipeline → Fabric data agent. "Which regions missed Q3 target, and by how much?" The data agent sits on the sales semantic model, writes the DAX, and only shows rows each person may see. Publish it into Microsoft 365 Copilot or Copilot Studio so leaders never open Fabric. Cold-chain monitoring for a food distributor → Fabric operations agent. Truck temperature sensors stream into an Eventhouse. The goal: "Keep chilled loads between 2 and 5 °C." When a truck drifts, the agent posts in the fleet Teams channel with the truck, route and trend, and can trigger a Power Automate flow to reroute or open a ticket. Retail "what should I do?" assistant → all four together. Microsoft's Zava reference demo shows this: an operations agent spots a stock risk, a Fabric data agent pulls the numbers, Foundry agents for sales and product reason over them, and Copilot Studio is the front door in Teams where the manager approves a reallocation. Better together The strongest designs use more than one. Copilot Studio meets people where they work, Foundry adds custom reasoning, and the two Fabric agents keep answers and actions grounded in governed data. The operations agent is the only one that starts on its own: it reads live events and reaches people in Teams without anyone asking first. Status as of October 2026, and gotchas Fabric data agent is GA, and its Copilot Studio integration went GA at FabCon Barcelona (October 2026). Data agents in Microsoft Copilot Chat were only a sneak peek there. The data agent inside Copilot in Power BI was retired on 26 August 2026. Use the data agent directly, or through Foundry, Copilot Studio, Microsoft 365 Copilot or its MCP endpoint. Operations agent is GA (June to July 2026). Newer abilities, such as root-cause investigations and pre-approved actions, are still in public preview. Operations agents need a paid Fabric capacity. Trial capacity is not supported, and an admin must turn on the Copilot/AI and operations agent tenant settings. A data agent is only as good as the model under it. Clear table names, descriptions, relationships and agent instructions matter more than prompt tricks. An operations agent automates your mistakes too. Start with "recommend" actions that a person approves, then move to automatic actions. Don't pick Foundry just for governance. Both Copilot Studio and Foundry agents now show up in the same Microsoft governance and security tooling. Pick on build style, model needs and network needs. Rule of thumb: prototype in Copilot Studio for a week. If you hit a wall (model choice, evals, isolation, scale), that wall is your Foundry requirement. Sources Operations agent (Generally Available), Fabric Updates Blog Improving operational efficiency with operations agents, Fabric Updates Blog Build, deploy, and govern Fabric at scale (FabCon 2026), Fabric Community FabCon Barcelona 2026 announcements, James Serra Retirement of data agent in Copilot in Power BI, Fabric Community What are Fabric data agents: GA status and limits, Atlan Fabric data agent vs operations agent, Daymark One agent, many channels: querying Fabric data agents, Data Science at Microsoft Copilot Studio vs Microsoft Foundry, Nasstar Zava Real-Time Intelligence demo, Microsoft CDXFabric Data Agents Get Smarter on 2 November. Can You Prove Yours Still Answers Correctly?
Microsoft announced Advanced NL2SQL and advanced DAX generation as generally available on 19 October 2026. The core improvements roll out to the standard runtime by default, together with a move to GPT-5.6 Terra. Separately, the old custom-tool way of calling a Fabric data agent from Microsoft Foundry stops being supported on 6 November. Date (2026) What happens What you should have done 19 Oct GA announced Read this post Now to 1 Nov Changes available on the preview runtime Build a ground-truth set, baseline standard, test preview 2 Nov Standard-runtime rollout begins, no opt-in Baseline frozen and stored 6 Nov Foundry custom-connection integration unsupported Foundry agents moved to Fabric IQ By 10 Nov Rollout expected to complete Re-run, diff the queries, tell affected users What Microsoft actually announced The query generators stop writing in one pass and start investigating first. That is why behaviour can change even where accuracy improves. Until now, NL2SQL and NL2DAX generated a query in a single pass, resolving schema, business definitions and filter values at once. As the GA announcement puts it, a wrong guess still runs, and the wrong answer looks like a right one. Advanced NL2SQL explores the schema, tests joins and looks up real filter values before writing SQL. It works in three modes: Example query following: adapts a relevant example query, for instance by swapping a filter value. Clarifying questions: asks the user when it can't answer reliably. Data exploration: infers logic from your examples and checks assumptions against the data. Advanced DAX generation brings the same idea to semantic models: an agentic tool-calling loop plus instance value indexing, so a user's "California" maps to the stored "CA". Microsoft's internal benchmarks Microsoft's 2026 internal evaluations, standard versus preview runtime. Microsoft notes results vary by workload, model and configuration. Benchmark Smart-answer accuracy P50 latency P95 latency NL2SQL, standard 78.8% 28.4 s 46.8 s Advanced NL2SQL, preview 90.9% 26.0 s 32.9 s NL2DAX, standard 89.15% 39.1 s 97.5 s Advanced DAX, preview 91.86% 20.4 s 34.7 s SQL gains 12.1 accuracy points. DAX gains only 2.7 points, but its tail latency drops by about 64%. If your semantic-model agents felt slow, that is your headline. What moves, and what doesn't Moving to standard: Advanced NL2SQL, advanced DAX generation and the GPT-5.6 Terra model upgrade. Staying in preview: schema descriptions and topics for SQL sources. Two details catch people out (runtime docs): A published agent keeps the runtime it was published with. Publish on preview and it stays on preview until you republish. Check every production agent. Runtime choice doesn't pick the model. A preview run today is a strong signal, not a guaranteed copy of standard after 10 November. Re-run after the rollout regardless. Why "better on average" can still surprise you A benchmark counts right answers. Your users notice changed answers. The second one generates support tickets. Workaround instructions start fighting the agent. Rules written to patch single-pass behaviour may no longer be needed, and unneeded guidance can steer an agent away from a path it would have found itself. Example queries get followed more literally. Good when they are current. An example with last year's fiscal logic now gets reproduced more faithfully. Ambiguous questions get a question back. Better for users, but it breaks anything downstream that expects a number, such as a Copilot Studio flow or a Foundry agent chain. A correct new answer can be a business problem. "Active customers in Q3" returned 41,200 because of a wrong date column; now it correctly returns 38,900. Someone may already have put 41,200 in a board pack. The pass count hides the churn. 40 of 50 pass before, 44 after. Looks like +4. Underneath, 6 newly pass and 2 now fail: 8 changed answers, 2 regressions. Question status Count Action Pass both, same query 34 Nothing Pass both, different query 4 Right for the right reason? Failed before, passes now 6 Tell anyone who acted on the old answer Passed before, fails now 2 Fix instructions or examples first Fail both 4 Existing backlog Illustrative numbers. The structure is what your own run should produce. The pre-rollout routine: six steps for this week You need a Fabric notebook, a lakehouse and the Fabric data agent SDK. The SDK and its evaluation features are in preview, so note the version you used. You also need F2 or higher capacity, the data agent tenant settings enabled and read access to the sources. Step 1: Inventory agents and dependencies For each published agent, record the runtime it was published on, its source types, where it is consumed (Fabric, Teams, M365 Copilot, Copilot Studio, Foundry) and its business owner. Anything called from Foundry via the old custom tool also goes on your 6 November list. Step 2: Build a ground-truth set that covers the risky cases The SDK takes a DataFrame or CSV with question and expected_answer. Aim for 30 to 60 questions per agent, ideally from real usage, covering filter values ("California" stored as "CA"), example-adjacent questions, multi-join questions, ambiguous questions and high-stakes KPI figures. %pip install -U fabric-data-agent-sdk import pandas as pd gt = pd.DataFrame(columns=["question", "expected_answer"], data=[ ["Total net sales in California for September 2026", "1,284,550.20"], ["Returns rate by region for FY26", "North 3.1%, South 2.7%, East 4.0%, West 3.4%"], ["What was revenue last year?", "CLARIFY: should ask gross or net, fiscal or calendar"], ["Active customers in Q3 2026", "38,900"], ]) gt.to_csv("/lakehouse/default/Files/agent_eval/ground_truth_sales.csv", index=False) The values are placeholders. Use answers your business owner has signed off. Step 3: Use a critic prompt that accepts clarifying questions The default grader would mark a good clarifying question as wrong. The SDK accepts a custom critic_prompt with {query}, {expected_answer} and {actual_answer} placeholders. critic_prompt = """ Query: {query} Expected Answer: {expected_answer} Actual Answer: {actual_answer} If the expected answer starts with CLARIFY:, respond 'yes' only if the actual answer asks a clarifying question about the same ambiguity. Otherwise respond 'yes' only if the actual answer is numerically and semantically equivalent. Is the actual answer equivalent to the expected answer? """ Step 4: Capture the standard baseline before 2 November Run against the production stage, which is what users hit today. Give each run its own table. from fabric.dataagent.evaluation import ( evaluate_data_agent, get_evaluation_summary, get_evaluation_details) agent_name = "SalesAgent" baseline_table = "eval_sales_standard_baseline_2026_10" baseline_id = evaluate_data_agent( gt, agent_name, table_name=baseline_table, data_agent_stage="production", critic_prompt=critic_prompt) print(get_evaluation_summary(baseline_table)) baseline = get_evaluation_details(baseline_id, baseline_table, get_all_rows=True) print(baseline.columns.tolist()) # check column names in your SDK version This writes <table_name> with per-question results and <table_name>_steps with the detailed reasoning and step-by-step execution. Look there for the generated SQL or DAX, and keep both tables as evidence. For a single question, the Diagnostics button exports the full steps too. Step 5: Record P50 and P95 latency through the MCP endpoint The evaluation tables focus on correctness, so time each question against the published agent's MCP endpoint. This adapts Microsoft's SDK sample: import time, numpy as np from azure.identity import InteractiveBrowserCredential from mcp import ClientSession from mcp.client.streamable_http import streamablehttp_client mcp_url = ("https://api.fabric.microsoft.com/v1/mcp/workspaces/<workspace-id>" "/dataagents/<data-agent-id>/agent") cred = InteractiveBrowserCredential() async def ask(q): tok = cred.get_token("https://api.fabric.microsoft.com/.default").token async with streamablehttp_client(mcp_url, headers={"Authorization": f"Bearer {tok}"}) as (r, w, _): async with ClientSession(r, w) as s: await s.initialize() tool = (await s.list_tools()).tools[0] arg = next(iter(tool.inputSchema["properties"])) t0 = time.perf_counter() await s.call_tool(tool.name, {arg: q}) return time.perf_counter() - t0 secs = [await ask(q) for q in gt.question for _ in range(3)] print(f"P50 {np.percentile(secs, 50):.1f}s P95 {np.percentile(secs, 95):.1f}s") Run each question several times; single timings are noisy. The MCP response returns answer text only, not the query, which is why query evidence comes from the steps table. Step 6: Preview now, standard after 10 November, then diff Preview, today. Switch your draft to preview and evaluate the sandbox stage, leaving production untouched. The switch comes from the runtime docs (the ribbon's Runtime dropdown works too); check the SDK reference for the FabricDataAgentManagement import. data_agent = FabricDataAgentManagement(agent_name) data_agent.update_configuration(enable_preview_features=True) preview_id = evaluate_data_agent(gt, agent_name, table_name="eval_sales_preview_2026_10", data_agent_stage="sandbox", critic_prompt=critic_prompt) data_agent.update_configuration(enable_preview_features=False) # don't publish on preview by accident Standard, after 10 November. Repeat Steps 4 and 5 against production with new table names. The diff. Compare per question, never just the summary. Fill in your SDK's pass-flag and answer column names. def diff_runs(before, after, passed="<pass-col>", answer="<answer-col>"): m = before.merge(after, on="question", suffixes=("_b", "_a")) def status(r): b, a = r[passed + "_b"], r[passed + "_a"] if b and a: return "same" if str(r[answer + "_b"]) == str(r[answer + "_a"]) else "answer changed" return "newly correct" if a else ("REGRESSION" if b else "fail both") m["status"] = m.apply(status, axis=1) return m.sort_values("status") Then compare the generated queries from both _steps tables. A changed query with the same answer deserves a look: it may be right by coincidence on this month's data. When a result changed, check the run steps for: which example query it picked, which join path it chose, which instruction it followed, and whether a clarifying question was the right call. Most fixes will be in your examples and instructions, not the platform. The 6 November deadline: move Foundry agents to Fabric IQ If a Foundry agent calls your data agent through the original custom tool, that path stops being supported on 6 November 2026 (retirement notice). The catch: the replacement, the Fabric IQ (OneLake Catalog) tool, is in preview. Production agents move from a deprecated integration to a preview one, four days into the query rollout. Make that an explicit risk decision, with a fallback plan. What you gain: pick agents by name from the catalog, connect several data agents to one Foundry agent (the old tool allowed one), and an MCP endpoint per agent. Microsoft also says responses will soon include the query used. Portal migration, in Microsoft's order: Publish the data agent with a rich, detailed description. In the Foundry portal, open the agent and add the Fabric IQ (OneLake Catalog) tool, then select your data agent. Update the Foundry agent's instructions to name each data agent and what it handles. Remove the old custom tool, save, and test with representative questions. For code, use the MCP URL from the agent's Settings > Model Context Protocol tab and the FabricIQPreviewTool in azure-ai-projects>=2.2.0. Full samples are in Connect agents to Microsoft Fabric with Fabric IQ. Test the two changes separately. If you migrate on 5 November and answers shift on 8 November, you won't know why. Migrate now and baseline through Foundry before 2 November. The connection calls Fabric as the signed-in user, so test with accounts that match real users' permissions and row-level security. When the new answer is right, who tells people? The hardest part is not technical. Someone acted on the old number. Decide now who owns that conversation: Each agent has a business owner who reviews "newly correct" and "answer changed" rows. High-stakes answers get a change note: old answer, new answer, why, and since when. The note goes where the old answer was used: report owners, Teams channels, Copilot Studio makers. Regressions get fixed, not avoided. Standard is changing anyway, so plan the fix. Also tell users before 2 November that answers may become more precise and the agent may ask follow-up questions. A warned surprise becomes a feature. Your checklist Inventory agents, runtimes, consumers and owners Build ground-truth sets, including ambiguous and high-stakes questions Write a critic prompt that accepts correct clarifying questions Capture the standard baseline, steps and latency before 2 November Run the set on preview in the sandbox stage Migrate Foundry agents to Fabric IQ and baseline them before 6 November Decide explicitly on running production on the preview Fabric IQ tool Re-run after 10 November and diff per question and per query Retire workaround instructions and refresh stale examples Send change notes for high-stakes answers that moved
