Description Microsoft Fabric strongly promotes concepts such as “Zero-ETL”, OneLake Shortcuts, Mirroring, and Direct Lake to reduce traditional ETL complexity. However, in real-world enterprise environments—especially hybrid architectures with on‑premises systems—ETL is often unavoidable, and this is where a critical gap appears. The Core Problem When Microsoft Fabric accesses on‑premises data through the On‑Premises Data Gateway (ODG): ODG can consume a large portion of on‑premises → cloud network bandwidth Neither Fabric nor ODG provides: Bandwidth throttling Rate limiting QoS or prioritization There is no first‑class observability for: Which pipelines or copy activities are responsible How much data they transfer When and how they saturate network links By contrast, Power BI semantic model refreshes are well covered by: Capacity Metrics Refresh history CU consumption visibility But Fabric Pipelines / Copy activities / Dataflows Gen2, which are often the largest network consumers, are not covered with comparable visibility. Why This Matters In enterprise networks: On‑premises bandwidth is shared with mission‑critical systems Uncontrolled ETL traffic can: Degrade core business systems Trigger operational incidents Block adoption of Fabric regardless of its analytical strengths Currently, customers are forced to: Infer ETL impact indirectly Manually correlate timestamps across multiple systems Treat ODG as a “black box” despite it being the critical choke point This is not a tooling inconvenience but a platform‑level design gap. Comparison with Other Platforms Other cloud data platforms explicitly address this concern: Databricks Provides pipeline‑level observability Supports rate limiting (e.g. maxBytesPerTrigger) Exposes network and throughput metrics as first‑class signals Snowflake Treats ingestion as a managed “entry point” Provides clear ingestion metrics and accountability per channel Microsoft Fabric, in contrast, effectively says: “Reduce ETL wherever possible,” but provides no adequate answers for unavoidable ETL. What This Idea Requests At minimum, Fabric should provide one or more of the following: Pipeline‑level observability for ODG traffic Data volume transferred Execution time Gateway‑level saturation indicators Correlation between Fabric Pipelines and ODG execution Even coarse, time‑window‑based correlation would help Optional throttling or rate‑limiting controls Per gateway Per pipeline or activity An official, supported monitoring story Instead of forcing customers to rely on log scraping or community workarounds Closing Thought “Zero‑ETL” is a valuable direction. But until ETL is truly eliminated, Microsoft Fabric must treat ETL traffic—especially via ODG—as a first‑class operational concern, not as something “outside the platform boundary.” Without this, Fabric adoption in serious hybrid enterprises will remain unnecessarily fragile.
... View more