Mirroring vs Sreaming in One Lake

Hi powerbiexpert22, thanks for the great questions. I'll try to go one by one:

Q: how mirroring is different from realtime ingestion like streaming?

Mirroring is a Fabric feature, that runs a background ingestion operation against a remote data source using a CDC or CDF protocol - where a remote data source publishes a feed of recent changes that the Fabric background process receives, analyzes and retrieves the data delta and then creates/updates a copy of the dataset on OneLake in a form of delta tables. Mirroring is a pull technique, because Fabric needs to pull actual delta from the remote data source using the change notifications.

Streaming, on the other hand, is a data source that produces a stream of messages, or events, that can be received by a Fabric component Eventstream that can also execute actions agains those messages, treating them as data. Messages can be structured like JSON or unstructured like text. Fabric Eventstream by itself does not query remote streaming source nor saves received messages anywhere in Fabric unless configured to do so. Eventstream can be thought of as a passive process compared to mirroring which is an active process.

Q: what does continuous data replication stands for in case of mirroring?

Continuous means that the mirrored data will continue to replicate to OneLake for as long as the connection between Fabric and the data source is not severed. Fabric will not stop the replication by itself.

Q: does continous mean zero latency or 1 or 2 seconds etc?

Continuous does not define specific latency. Latency depends on many factors most of them external to Fabric and related to how often the remote data source publishes change notifications and how fast a connection is between Fabric and remote data source. See the next questions for more.

Q: what is the frequency of data replication in case of mirorring?

Because Fabric runs a background pulling process there is a predefined frequency for how often the change feed can be processed, but it's not configurable nor documented for the time being. From practical experience, a remote SQL server mirrored in Fabric would update in a window of 2-5 minutes after the change has happened in the remote dataset.

Q: in which scenario we should use mirorring rather than ingesting data using pipelines?

Fabric mirroring is design to implement a "Zero-ETL" approach where fast implementation and low maintenance is more improtant than actual latency or additional factors such as data transformations or quality gates. In addition, Fabric mirroring requires certain conditions to work reliably. If one needs a more controlled replication, more complex ingestion logic, or mirroring prerequisites cannot be met, Fabric mirroring may not be an ideal solution and alternatives like data pipelines or Notebooks should be used. There are also thrid party tools to implement continuous fully controlled low latency ingestion that work with Fabric directly such as Striim. On the other hand, Fabric mirroring is available for free with every F-Sku capacity so it should be considered a first choice when ingesting from supported data sources.

If you found this answer helpful please consider giving kudos and mark as a solution to help other community members.