Forum Discussion
Internal or external Delta Table? Recommendations
I’m planning to start building a data counter using the Medallion architecture. I’m wondering whether it’s better to load Delta tables as internal or external.
Any recommendations based on your experience (performance, storage, etc.)?
Hi elizabetars8703 ,
If you are new to Medallion architecture in Microsoft Fabric, internal (managed) Delta tables are generally recommended. They offer better performance, simpler lifecycle management, and seamless integration with OneLake. External tables are useful when you need cross-platform access or must keep data outside Fabric’s managed environment.
You are trying to build a data counter with Medallion architecture, here’s a practical approach:
1. Start with internal Delta tables for Bronze, Silver, and Gold layers. This ensures optimal performance and easier management.
2. Use external Delta tables selectively if you need to integrate with existing data lakes or share data across platforms.
3. Consider storage costs and governance: internal tables simplify governance in OneLake, while external tables may require additional policies and monitoring.
Note: External tables can introduce latency and complexity in schema management. Internal tables lock you into OneLake; if you need portability across clouds, external may be better.
Hybrid approach is common: internal for curated layers (Silver/Gold), external for raw ingestion (Bronze). Internal Delta tables are the recommended default. Use external tables only when you have strong reasons (compliance, cross-platform, or existing external data lakes).
If my response as resolved your issue please mark it as solution and give kudos.
4 Replies
Hi elizabetars8703 ,
If you are new to Medallion architecture in Microsoft Fabric, internal (managed) Delta tables are generally recommended. They offer better performance, simpler lifecycle management, and seamless integration with OneLake. External tables are useful when you need cross-platform access or must keep data outside Fabric’s managed environment.
You are trying to build a data counter with Medallion architecture, here’s a practical approach:
1. Start with internal Delta tables for Bronze, Silver, and Gold layers. This ensures optimal performance and easier management.
2. Use external Delta tables selectively if you need to integrate with existing data lakes or share data across platforms.
3. Consider storage costs and governance: internal tables simplify governance in OneLake, while external tables may require additional policies and monitoring.
Note: External tables can introduce latency and complexity in schema management. Internal tables lock you into OneLake; if you need portability across clouds, external may be better.
Hybrid approach is common: internal for curated layers (Silver/Gold), external for raw ingestion (Bronze). Internal Delta tables are the recommended default. Use external tables only when you have strong reasons (compliance, cross-platform, or existing external data lakes).
If my response as resolved your issue please mark it as solution and give kudos.
Thank you so much Anonymous
- For the Gold Layer, do you recommend using a Delta Table or a Warehouse?
Hi elizabetars8703 ,
If your Gold layer is primarily for BI dashboards and reporting --> Use a Warehouse.
If your Gold layer must serve both BI and ML/data science --> Consider a hybrid:
Store curated data in Delta tables for openness.
Expose a Warehouse view for business consumption.
This hybrid approach is common: Lakehouse for flexibility, Warehouse for performance and governance.
Refer below links.
https://learn.microsoft.com/en-us/fabric/onelake/onelake-medallion-lakehouse-architecture
https://learn.microsoft.com/en-us/fabric/data-warehouse/guidelines-warehouse-performance
https://learn.microsoft.com/en-us/fabric/fundamentals/decision-guide-lakehouse-warehouse
I hope this information helps.
Regards,
Dinesh
