Forum Discussion

db042190's avatar
db042190
Impactful Individual
16 days ago

getting started on medallion

hi im almost thru a course on fabric.   trying to align what im hearing with the medallion architecture.   we want to plumb/ingest s4 tables at the lowest levels (maybe with a little planned redundancy) into bronze.   i like the idea of using what i hear is parquet (maybe aka delta) at that level.  but i get a little confused by all the choices thereafter.   i think landing silver in parquet would be nice but suspect sql server is an option as well as all these other things i hear about like data warehouse etc.   then gold , sql server, data warehouse, vldb, maybe even direct lake.   finally, my peers think they want api's AFTER bronze, AFTER SILVER and AFTER GOLD, maybe supported by an api gateway (whatever that is).   imho, if u r landing something in bronze you are planning on moving it along to the other layers.   and other than various layers being in sql server, i dont see how apis in the traditional sense would be useful there.   ive heard of ksql and using notebooks interactively on those layers during dev but APIs?

7 Replies

  • Hi db042190​,

    It sounds like you have a lot more to understand about Fabric before implementing an end to end system. 

    Let me help define a few things: 

    Fabric data stores: 
    There are 4 main data stores in Fabric: 
    Lakehouse
    Warehouse
    KQL Database
    SQL Database

    A Lakehouse supports structured and unstructured data, and behind the scenes stores structured data in Delta Parquet files in OneLake, and unstructured data in ADLS Gen 2 storage in OneLake. You don't need to interact with the parquet files, Fabric does that for you. 
    The main ways to interact with a Lakehouse are with Spark (PySpark or SparkSQL) for writing, and there is a read only SQL endpoint (T-SQL) for consumption. 

    A Warehouse only stores structured data, and is transactional. Under the hood it also uses Delta Parquet files in OneLake. T-SQL is the main way to read and write to a Warehouse. 

    KQL database are for real time intelligence, this doesn't seem to be what you're doing so I won't go into depth here. 

    SQL Databases are essentially Azure SQL sitting on Fabric compute. This gives you the full functionality of SQL Server. Data is stored in the proprietary SQL Server format, and then mirrored into OneLake in Delta Parquet files. Data is written and consumed at the SQL Server level with T-SQL, and data can be read from the OneLake Mirror level using Spark or SQL, but as this is a mirror, this is read only by nature. 

     

    You also mentioned the gateway. The only gateway in Fabric is the On Prem Data Gateway (Or the Azure Virtual Network Gateway, same thing as far as this explanation goes). This is used to connect Fabric to your on prem data sources. It acts as the bridge between your company's network and Fabric. 

     

    When designing a medallion architecture in Fabric, here are some considerations: 
    1: What will be your data store for each layer and why? 

    • Will you use Lakehouses, Warehouses, or SQL Databases? What are the benefits of your decision for your use case? 

    2: Where is your source data coming from? (If it's coming from an on prem database, you will need to set up a gateway) 
    3: How will users consume your data? (Will they query the gold database, will they only consume it in reports/semantic models, or will they want another consumption method like a GraphQL API?) 

    • db042190's avatar
      db042190
      Impactful Individual

      thx amy, yes we know about and already have our On Prem Gateway.  In fact we have 2 temporarily after our big acquisition and have recommended failover on the one single one from before the merger .  My peers are looking at something called an "api gateway" which i'll post more info about later after ive taken a closer look.   it may have everything to do with apis and possibly is not limited to pbi/fabric specific technologies.   but my peers intend for it to help with consumers (software/people etc) reading any layer of the medallion architecture and also logging usage by acct.   i dont know why they'd want to read anything but gold or sm but we'll see.  i'm looking up GraphQL API now.

  • Hi db042190​ 

    Further to what has been said, I would also look at the way your business functions and works and sometimes you might not need all three layers. If you're getting data that is really clean and structured, that potentially could be your silver layer where you just need to do the transformation into a star schema into your gold model. So it often depends on your data and ways of working in your organization.

  • ShivekMaharaj's avatar
    ShivekMaharaj
    Power Participant

    Hi db042190​,

    tayloramy​ has covered the Fabric stores well. One thing I would add is that medallion is primarily about data quality and purpose, rather than choosing a different Fabric storage technology at every layer.

    Microsoft's Fabric medallion guidance actually documents both an all-Lakehouse approach and a pattern where Bronze and Silver are Lakehouses and Gold is a Warehouse.

    For a first implementation, I would keep it simple:

    • Bronze: Lakehouse, raw source data
    • Silver: Lakehouse, cleaned/conformed Delta tables
    • Gold: Lakehouse or Warehouse, depending on the consuming workload


    Also, Parquet and Delta are related but not the same thing. Fabric Lakehouse tables use Delta by default. The data is stored in Parquet files underneath, while Delta adds the transaction log and table metadata.

    I would not introduce a Fabric SQL Database into Silver or Gold purely because it is available. Microsoft's data-store guidance positions SQL Database primarily for transactional/operational workloads, Warehouse for SQL-based analytical warehousing, and Lakehouse for data engineering and medallion scenarios.

    On the API side, I also would not put an API between Bronze, Silver and Gold unless there is an actual consumer that needs one. Those layers can flow internally using pipelines, notebooks, Spark or SQL.

    If applications need programmatic access to curated data, Fabric API for GraphQL can expose selected objects from a Lakehouse, Warehouse or Fabric SQL Database. I would normally expose Gold that way, and perhaps Silver for specific technical consumers, while keeping Bronze internal.

    So I would start with the workload requirements first, then choose the Fabric store for each layer rather than trying to use every available Fabric technology.

    AI-assisted drafting: AI was used to help structure and phrase this response. I reviewed and validated the technical content before posting.

    • db042190's avatar
      db042190
      Impactful Individual

      thx shivekmaharaj.   its not exactly api's BETWEEN say bronze and silver but coming out of one or more of the layers.   i listened a bit to an explanation yesterday and it sounded like one peer was saying due in part to governing people going to the well too often.  never heard of api's being used that way but am listening as much as possible to what they are saying.

  • v-aatheeque's avatar
    v-aatheeque
    Community Support

    Hi db042190​ 

    Have you had a chance to look through the responses shared earlier? If anything is still unclear, we’ll be happy to provide additional support.

  • v-aatheeque's avatar
    v-aatheeque
    Community Support

    Hi db042190​ 

    Following up to confirm if the earlier responses addressed your query. If not, please share your questions and we’ll assist further.