Using the Mirroring feature in a data warehouse project

[pmscorca]

Hi,

I need to design a data warehouse to feed with data coming from some database sources.

Could be a good thing using the Mirroring feature at bronze layer instead of implementing a lakehouse in order to accelerate the raw ingestion?

Thanks

[DataBard]

When you refer to lakehouse, I am assuming you mean lakehouse shortcuts. If I am incorrect on that assumption, please clarify.

Since the data would be replicated and delta tables created in OneLake, there's a chance you will get better performance from mirroring when ingesting your data into your silver layer.

For each data source, be sure to check that the source is supported by current shortcut/mirroring options. In most situations, your source would either be supported by shortcuts or Mirroring, but not both. Also be sure to check that your source is supported by mirroring, as current mirroring options focus on specific sources and have limitations around networking.

If the question is just 'Is Mirroring good for a bronze layer instead of writing processes to put it in a lakehouse?', I'd say the answer is yes. You'll save significant dev time mirroring your data than building Copy Jobs or similar processes to do the same. Just be sure to check performance impact on your source environments, as Mirroring is essentially performing CDC operations on your source data.

If this post helps, please consider accept as solution to help other members find it more quickly.

[pmscorca]

Hi,

when I talk about a lakehouse I refer to the object and not to the related shortcut.

I'd like to understand if Mirroring feature could be really useful in a medallion architecture to implement a data warehouse: it seems to be a valid alternative respect to a lakehouse in the bronze layer; so, Mirroring feature could be an accelerator component to deliver a data warehouse solution, isn't it?

Next, at silver layer I could have a lakehouse that reads from the bronze mirrored data source and so on.

I've already read the documentation about this feature.

Thanks

[DataBard · Accepted solution]

Yes, Mirroring is a great accelerator for bronze if you want that layer to be raw, untransformed data from your source environment. I've used mirroring in a medallion architecture with good success. At that point it's Delta tables sitting in your OneLake, which can be consumed for processes to move your data into silver.

[dbWizard]

My issue is when mirroring breaks and requires full rebuild. Then all the downstream dependencies fail. I'm thinking I mirror data to fabric, then have a process to immediately copy that data into a lakehouse to provide a buffer and layer of redunancy while items can be pointed at the lakehouse instead of mirror database so in the even that mirroring breaks we can keep on trucking while mirroring gets fixed. Thoughts?

[pmscorca]

Hi, thanks for your reply.

It seems that Mirroring supports only some data sources. Could I use Mirroring for an Oracle or MySQL source? Open mirroring?

Thanks

[DataBard]

In current state, you'd have to use Open Mirroring for Oracle or MySQL.

[Anonymous]

HI @pmscorca,

I'd like to suggest take a look at the document of mirroring feature at first:

Mirroring - Microsoft Fabric | Microsoft Learn

Regards,

Xiaoxin Sheng