How are teams structuring Microsoft Fabric Warehouse for scalable ELT and reporting workloads?
I’m looking at architecture patterns for Microsoft Fabric Warehouse in environments where multiple data sources, transformation jobs, and Power BI workloads all depend on the same platform. A typical flow might look like: Source systems → ingestion → staging → transformations → Fabric Warehouse → semantic models → Power BI I’m particularly interested in how teams are handling: staging vs curated schemas ELT transformations inside the Warehouse incremental loading patterns data quality checks before publishing CI/CD across dev, test, and production workload separation for engineering vs reporting monitoring failed loads and long-running queries deciding when to use Warehouse vs Lakehouse for the same solution For teams running Fabric Warehouse at scale, what architecture patterns have worked best for keeping the platform maintainable and avoiding duplicated transformations?
How are teams using AI agents to automate modern data warehouse workflows?
I am exploring how AI agents can help improve data warehouse operations by automating repetitive tasks and assisting data teams with faster decision-making. Some areas I am interested in: Automating data pipeline monitoring and issue detection AI-assisted SQL generation and optimization Automatically identifying data quality issues Triggering workflows based on business events or data changes Generating documentation and insights from warehouse metadata With platforms like Microsoft Fabric Data Warehouse, how are teams approaching AI integration? Are you using: Fabric pipelines with AI-powered automation? Copilot or LLM-based assistants for warehouse development? Custom AI agents connected with data warehouse APIs? Would love to hear real-world architecture patterns and best practices from data engineers working with Fabric.
Best approach(es) to refresh Dev Warehouse from Prod Warehouse (Read-Write Copy) in Microsoft Fabric
We are currently evaluating the best practices for refreshing a Development Data Warehouse using data from a Production Data Warehouse within the same Fabric tenant. We have evaluated a few approaches based on the use case: Read-Only Access (Zero-Copy): If Dev only needs read access to Prod data, using OneLake Shortcuts is the clear choice. It provides instantaneous access with zero data duplication, no storage overhead, and minimal setup effort. Read-Write Access (Full Schema & Data Copy): If Dev requires a Read-Write copy (allowing developers to run DML operations like UPDATE, INSERT, or DELETE without impacting Production), our current options seem to be: Fabric Data Pipelines (Copy Activity): Scheduled or trigger-based pipelines to pull data from Prod to Dev. Cross-Database Querying / CTAS: Running T-SQL (CREATE TABLE AS SELECT ...) across workspaces. The Challenge: Both Read-Write options (Pipelines and CTAS) can become cumbersome to set up and maintain, especially as datasets grow. Additionally, duplicating large volumes of data incurs extra storage and CU (Capacity Unit) processing costs. Was wondering if there are better, more efficient options available in Microsoft Fabric to maintain or periodically refresh a Read-Write copy of a Warehouse in a Dev workspace? Is Table Cloning / Zero-Copy Clone supported or planned for cross-workspace scenarios? How are others managing periodic Dev environment refreshes from Prod without creating heavy data copy pipelines?Fabric pipeline Deployment dev to prod
Hi there, I have been developing fabric data pipelines, warehouse, lakehouse in a fabric dev workspace. I noticed when deployment is done from WS_Dev to WS_Prod, pipelines are still pointing to Dev workspace. I can parameterize it. Where in Prod workspace do I need to configure that all objects should point to Prod workspace so code deployments from dev dont overwrite the variable in PROD. master pipeline with look up After lookup -- invoke pipeline child pipeline with copy activity.
