Fabric Snowflake connector returns "Invalid credentials" with Key-Pair despite valid key pair
The Fabric Snowflake connector fails with "Invalid credentials" when using Key-Pair authentication, in both Dataflow Gen2 and the Copy job. The same key pair authenticates successfully outside Fabric (Snowflake Python connector returns a valid session), which proves the credentials, private key, username, account, and warehouse are all correct. The failure occurs both on the cloud connection and through an on-premises data gateway (routing via a different network), which rules out a network/OCSP cause and points to a defect in the connector's key-pair handling. Environment Snowflake trial account UAE Region DESC USER confirms HAS_KEYPAIR = TRUE, RSA_PUBLIC_KEY_FP populated, no MFA, no auth/network policy Fabric region: UAE, Connector Implementation 2.0, on-prem data gateway (latest version) Steps to reproduce Create a Snowflake connection (Dataflow Gen2 or Copy job), Authentication kind = Key-pair, upload unencrypted PKCS#8 private key. Server tried in both forms: XXXXX-XXXXXX.snowflakecomputing.com and XXXXX.uae-north.azure.snowflakecomputing.com. Result: "Invalid credentials." Same result with an on-premises data gateway. Proof credentials are valid The identical private key + account + user authenticate successfully via snowflake-connector-python (returns CURRENT_USER=myuser, CURRENT_ACCOUNT=myaccount, CURRENT_ROLE=ACCOUNTADMIN). Data was successfully loaded Snowflake → Lakehouse via a Fabric Notebook using the same key. Expected: Key-pair connection succeeds (as it does in every non-Fabric client). Actual: "Invalid credentials" in both native connectors, cloud and gateway. Screenshots: Dataflow Gen2 connection dialog, Key-pair + gateway + "Invalid credentials": 2. Copy job connection dialog — Key-pair + gateway + "Invalid credentials": 3.DESC USER MELSHEIKH showing HAS_KEYPAIR = TRUE / RSA_PUBLIC_KEY_FP Is there any known issue regarding Fabric Dataflow Gen2 and Fabric Copy job connection to Snowflake?
Immense SQL endpoint lag with external data share
Hello everyone, our setup is: Tenant A: Mirroring a Business Central database into Fabric, external data share into Tenant B Tenant B: external data share shortcut landing in lakehouse (arrival in lakehouse works fine within a minute), DBT jobs access data via SLQ (that's why we're quite insistent on using SQL, DBT jobs are great) Our problem is: We're no strangers to SQL endpoint lag, but it is reaching unsustainable levels. Refreshing via the "Sync Metadata" button in the SQL endpoint GUI takes multiple hours. Without refresh the data becomes stale, up to several days old. Tables appear in the SQL endpoint GUI table list but are unreachable ("not existing") via SQL queries. Pipeline refresh activity and refresh via notebook take less but still too much time. Our capacity load is well within the limits, that's no probable cause. Our questions would be: A simple fix or workaround is always welcome, but also Is this a specific/known problem with external data shares (in combination with SQL endpoint)? Shortcuts should be a simple, latency free thing, right?
Fabric Workspace Architecture
I'm a Data Analytics Engineer at a logistics startup, and I'm the only person handling the data space. I'm looking for insights on how to effectively manage and monitor ETL processes using metadata-driven pipelines for a medallion architecture. I have 10 applications with about 2k tables total, though NetSuite accounts for 40% of the data. I'm considering 3 approaches for the Fabric workspace architecture: Approach 1: Create a dedicated NetSuite workspace with bronze, silver, and gold pipelines, plus 3 lakehouses for the medallion layers and schemas for individual departments. Then create similar workspaces for other applications. Approach 2: Organize by layer instead—separate workspaces for raw-to-bronze, bronze-to-silver, and silver-to-gold, with each workspace handling all applications at that layer. Approach 3: Use a single workspace to manage all medallion layers for all applications, but I'm not sure how to easily manage and monitor all the pipelines in this setup. Can you give me the pros and cons for each approach and recommend which one is best for a solo data engineer?Enable mermaid diagrams rendering in Notebooks
Diagrams can be a great explainer of a (Data engineering) process, code or ML process. Notebooks already support including Markdown for adding description. And Mermaid is a well-know way to include diagrams in markdown as text. It's used by GitHub. It does not work in Fabric Notebooks, and that would be a great addition. Workaround: Use services like mermaid.live, but then the content of your diagram "travels" outside Fabric tenant.
Fabric Capacity Metrics - missing subscription data
Good morning, From July 25 I cannot see subscription CU consumption in the App (North Central US region) - does anyone else have this issue? Any hints what is going on? I've raised MS support ticket but they keep on asking for random screenshots for some time now. It seems that some visuals include that data, some of them don't. Tomasz
Monitor your Eventstreams with Workspace Monitoring — Now with per-Eventstream control (Preview)
When a real-time data pipeline breaks, the first question is always the same: what went wrong, and where? Eventstream observability in Microsoft Fabric brings that visibility directly into Workspace Monitoring, so you can track the health, performance, and errors of your eventstreams without leaving the Fabric portal or setting up external monitoring infrastructure. With this update, eventstream observability returns with a redesigned approach that puts you in control — including a per-eventstream toggle that lets you decide exactly which eventstreams emit monitoring data and which don't. Why observability matters for real-time pipelines Real-time data pipelines operate continuously. Unlike batch jobs that run and finish, an eventstream can process data for days or weeks without anyone checking on it. When something goes wrong — a destination write error, or backlogged events — the problem often goes unnoticed until downstream consumers report stale or missing data. Eventstream observability closes that gap. When enabled, each eventstream emits structured metrics and error data to your workspace's monitoring Eventhouse. This data lands in three dedicated KQL tables that you can query, alert on, and build dashboards against: Table What it captures Emission frequency When to use it EventStreamMetrics Throughput, event counts, ingress and egress rates ~1 minute Capacity planning, data flow validation, throughput monitoring EventStreamErrorMetrics Error category, error code, severity, affected events ~1 minute Troubleshooting failures, identifying recurring error patterns EventStreamNodeStatus Node state (running, stopped) ~6 hours Health checks, topology status overview Once monitoring is enabled for an eventstream, these tables appear in your workspace's monitoring KQL database alongside existing Eventhouse and other workload tables. Figure: Eventstream monitoring tables in the Workspace Monitoring KQL database, showing EventStreamMetrics, EventStreamErrorMetrics, and EventStreamNodeStatus tables available for querying. What changed: Per-eventstream control During the initial preview, eventstream observability was enabled globally for all eventstreams in a workspace. Based on customer feedback, the feature now provides granular control — you choose exactly which eventstreams emit monitoring data, giving you flexibility to focus observability on the pipelines that matter most. The key changes: Per-eventstream toggle. A new setting — Log Eventstream activity — appears in each eventstream's settings panel. You can enable or disable observability for each eventstream individually, so a workspace with ten eventstreams can monitor only the two or three that are most critical. Default OFF. All eventstreams default to monitoring OFF. You choose which ones to enable. How to get started Eventstream observability requires Workspace Monitoring to be enabled on your workspace. If you haven't set it up yet: Open your workspace settings. Navigate to the Monitoring section. Enable Workspace Monitoring and wait for the monitoring Eventhouse to provision. Once Workspace Monitoring is active: Open any eventstream in your workspace. Select the Settings gear icon. Find the Log Eventstream activity toggle. Enable it. Within a few minutes, data begins flowing into the eventstream tables in your monitoring KQL database. You can query them directly with KQL, build Real-Time Dashboards, or use KQL querysets for deeper analysis. What comes next This release covers performance metrics, error metrics, and node health — the foundation for operational observability. Additional capabilities are planned for future updates: Diagnostic logs — Error logs from the ASA processing engine, providing deeper root-cause analysis for failures like deserialization errors, runtime query errors, and output write failures. Expanded connector coverage — Monitoring support for connector-based sources is under investigation and will be added in a future phase. Next steps Refer to the Eventstream monitoring in Workspace Monitoring documentation to learn more about configuration and supported tables. Refer to the Workspace Monitoring overview documentation for setup instructions. Share your feedback and feature requests in the Fabric Community Forums. Explore the Create a Real-Time Dashboard documentation to build live dashboards from your monitoring data.
Fabric Lakehouse – Is there a limitation on concurrent Livy/Spark JDBC sessions?
Hi, I am working on an external Java integration with Microsoft Fabric Lakehouse where I need to perform CRUD operations on Lakehouse Delta tables. I created a standalone Java application using the Microsoft Fabric Spark JDBC driver (ms-sparksql-jdbc), which internally uses the Fabric Livy/Spark session. The basic CRUD operations are working successfully. For example, I can execute an INSERT like: Class.forName("com.microsoft.spark.livy.jdbc.LivyDriver"); String connectionUrl = String.format( "jdbc:fabricspark://api.fabric.microsoft.com;" + "FabricWorkspaceID=%s;" + "FabricLakehouseID=%s;" + "AuthFlow=5;" + "AuthAccessToken=%s;" + "LogLevel=INFO;" + "ReuseSession=true", WORKSPACE_ID, LAKEHOUSE_ID, accessToken ); Connection conn = DriverManager.getConnection(connectionUrl); String sql = "INSERT INTO snaplogic_test_customers " + "VALUES (?, ?, ?, ?)"; try (PreparedStatement pstmt = conn.prepareStatement(sql)) { pstmt.setInt(1, 1); pstmt.setString(2, "Alice Johnson"); pstmt.setString(3, "[email protected]"); pstmt.setDouble(4, 1250.50); pstmt.executeUpdate(); } Issue I am seeing When I run the standalone application for the first time, it works successfully. In the Fabric Monitoring Hub, I can see a Fabric -JDBC- session being created, and the CRUD operation completes successfully. However, if I run the same Java application again immediately after the previous execution completes, the second execution gets stuck during: Connection conn = DriverManager.getConnection(connectionUrl); The application remains stuck at the connection creation stage. Interestingly, if I wait approximately 5–10 minutes and then run the application again, it works successfully. My questions Is there any limitation on the number of active/concurrent Livy/Spark sessions for a Fabric Lakehouse? Is there a limitation where only one regular Livy/Spark session can actively run against a particular Lakehouse/workspace/capacity at a time? If multiple external applications need to access the same Lakehouse concurrently, are multiple Livy/Spark sessions expected to work in parallel? If multiple sessions are supported, what could cause the second call to:DriverManager.getConnection(connectionUrl) to wait until the previous JDBC/Spark session is completed or released? I need multiple external applications or requests to perform CRUD operations concurrently on Fabric Lakehouse tables, what is the recommended approach? My use case is to build an external application where multiple requests may need to perform SELECT, INSERT, UPDATE, and DELETE operations on Lakehouse tables concurrently. I am currently using the Fabric Livy API/Spark SQL approach. I want to understand whether there is a better or recommended Fabric-supported approach for implementing concurrent CRUD operations against Lakehouse tables. Any guidance or recommendations on the best approach would be greatly appreciated. Thanks in advance! Regards, BalramMissing Dynamics 365 F&O tables when creating a Link to Microsoft Fabric
Hi Community, We are creating a Link to Microsoft Fabric from our Dataverse environment. While selecting tables for the link, we can see all the required tables under the Dataverse section. However, under the Dynamics 365 Finance and Operations (F&O) section, I see lot of tables, but required F&O tables are not available for selection when i search for it. Could you please clarify the following? Are there any features, configurations, or prerequisites that must be enabled for an F&O table to appear when creating a Link to Microsoft Fabric? Do the underlying F&O tables need to have specific properties enabled, such as change tracking or any other data-sharing configuration? Are there any restrictions on which standard or custom F&O tables are supported? Is there a recommended way to verify why a specific F&O table is unavailable? The required tables exist in the F&O environment, but they are not listed in the F&O table-selection section while configuring the Fabric link. Any guidance on the required configuration or troubleshooting steps would be appreciated. Thanks.
Column value masking for OLS
When applying Object-Level Security (OLS) in Power BI using columnPermission = none, the protected column is completely unavailable to the user. As a result, if that column is used in an existing visual, the visual may break or display an error. It would be useful to have an additional column-level permission that masks or suppresses the column values without removing the column itself from the semantic model. This way if the column (or measure) is available in a visual it would not break but simply it's value will not be visible for roles who do not have permissions on the column. This way there is no need to make separate visuals (or separate report) for every role. Any measure built using such column (or measure) should carry this permission rendering the resultant measure as masked. This would be similar to the OMIT functionality available in Qlik, where sensitive values can be hidden without negatively affecting the overall user experience.
