Forum Discussion
getting started on medallion
hi im almost thru a course on fabric. trying to align what im hearing with the medallion architecture. we want to plumb/ingest s4 tables at the lowest levels (maybe with a little planned redundancy) into bronze. i like the idea of using what i hear is parquet (maybe aka delta) at that level. but i get a little confused by all the choices thereafter. i think landing silver in parquet would be nice but suspect sql server is an option as well as all these other things i hear about like data warehouse etc. then gold , sql server, data warehouse, vldb, maybe even direct lake. finally, my peers think they want api's AFTER bronze, AFTER SILVER and AFTER GOLD, maybe supported by an api gateway (whatever that is). imho, if u r landing something in bronze you are planning on moving it along to the other layers. and other than various layers being in sql server, i dont see how apis in the traditional sense would be useful there. ive heard of ksql and using notebooks interactively on those layers during dev but APIs?
3 Replies
Duplicate of
getting started on medallion | Microsoft Fabric Community
Hi db042190 ,
It sounds like you have a lot more to understand about Fabric before implementing an end to end system.
Let me help define a few things:
Fabric data stores:
There are 4 main data stores in Fabric:
Lakehouse
Warehouse
KQL Database
SQL Database
A Lakehouse supports structured and unstructured data, and behind the scenes stores everything in Delta Parquet files in OneLake. You don't need to interact with the parquet files, Fabric does that for you.
The main ways to interact with a Lakehouse are with Spark (PySpark or SparkSQL) for writing, and there is a read only SQL endpoint (T-SQL) for consumption.A Warehouse only stores structured data, and is transactional. Under the hood it also uses Delta Parquet files in OneLake. T-SQL is the main way to read and write to a Warehouse.
KQL database are for real time intelligence, this doesn't seem to be what you're doing so I won't go into depth here.
SQL Databases are essentially Azure SQL sitting on Fabric compute. This gives you the full functionality of SQL Server. Data is stored in the proprietary SQL Server format, and then mirrored into OneLake in Delta Parquet files. Data is written and consumed at the SQL Server level with T-SQL, and data can be read from the OneLake Mirror level using Spark or SQL, but as this is a mirror, this is read only by nature.
You also mentioned the gateway. The only gateway in Fabric is the On Prem Data Gateway (Or the Azure Virtual Network Gateway, same thing as far as this explanation goes). This is used to connect Fabric to your on prem data sources. It acts as the bridge between your company's network and Fabric.
When designing a medallion architecture in Fabric, here are some considerations:
1: What will be your data store for each layer and why?- Will you use Lakehouses, Warehouses, or SQL Databases? What are the benefits of your decision for your use case?
2: Where is your source data coming from? (If it's coming from an on prem database, you will need to set up a gateway)
3: How will users consume your data? (Will they query the gold database, will they only consume it in reports/semantic models, or will they want another consumption method like a GraphQL API?)Hi db042190
We would like to inquire whether have you got the chance to check the solutions provided by tayloramy in community to resolve the issue. We hope the information provided helps to clear the query. Should you have any further queries, kindly feel free to contact the Microsoft Fabric community.
Hi db042190
We haven’t heard from you on the last response and was just checking back to see if you have a resolution yet. And, if you have any further query do let us know.
Thank you.
