Reinventing the Gold Layer Using Materialized Lake Views

hi pallavi_r , AparnaRamakris 
Thanks for your responses. Much appreciated.

I think the fact that each of you had a somewhat different opinion on how/where MLVs could be used serves to highlight the concern/confusion here. 

AparnaRamakris : I did read your link, and I will comment on the DQ report, but wanted to lay out some thoughts first.

If we consider the DQ checks to be business rules, then any issues with the data need to be actionable. There are two aspects here that concern me; DRY (do not repeat yourself) and alerting. 

Ideally a business rule should not need to be repeated in multiple places (DRY). Doing so creates a double maintenance burden and the risk of issues/bugs if various copies do not agree. 
With the both DROP and FAIL in order for them to be actionable some (or all) of the incorrect rows need to be identified.  Obviously we can issue a query to find them but...this query has to repeat the business rules from the DQ checks and get it right. For instance if there are three checks A, B, C then the query needs to be NOT (A or B or C). This repetion is a risk/burden whether being done manually by a troubleshooter or embedded in some further automation. 

There is also the (smaller) matter that (presumably) both the DROP and FAIL DQ checks have identified specific incorrect rows (all for DROP, or at least one for FAIL ?) and used some compute to do so but we then have to expend further compute to explicitly find them "again" . In general storage is orders of magnitude cheaper than compute so having the first result (failed rows) stored , would be preferable to expending compute to get them again. Or at least have the option for them to be stored.

The next thing is alerts.

For FAIL presumably it is fairly easy to set up an alert as an explicit error occurs. With DROP it is less clear how to achieve this (likely activator triggers off the DQ report underlying semantic model). In both cases the repetition of rules/compute above would be seem still be needed for the troubleshooting and maybe even to generate the alert. 

> According to me ,this is more than enough to backtrack on the bad record
> and make sufficient corrections on them from business if needed.

I have not had much time yet to dig into what is in the DQ report, but unless it does actually contain the details or ids of all the failed rows (i.e. they _are_ stored) the duplication mentioned above will still be required. 
I have not taken a close look at the underlying semantic model and assessed whether there is anything that can be used to automate alerting / troubleshooting / reprocessing. I suspect it will only contain aggregates at per refresh granularity. This would be suffient for alerting if it can be hooked up to Activator but would mean actioning would still require the duplication mentioned above. Conversely if there was the option to quarantine failed rows to storage this would not only address the duplication issue but could also likely be used to trigger automation.

Now by saying this I am not saying that MLVs are useless or lack value. It is more that it makes it unclear or confusing when/where it is reasonable to use them given realistic operational requirements. So for example I see some articles indicating use in scenarios where input data is already relative clean (silver, gold) but others that indicate value earlier but with no real indication of how to address the operational matters I have mentioned. 


To some extent the DQ checks almost feel like a feature that can't really be used i.e. if you sidestep the issue by saying that it is only suitable for use on relatively clean/curated data then it this seems to be equivalent to saying only use DQ checks in situations where they probably won't be needed, or if you do say use them on dirtier data then the fact that you will probably have to create a duplicate WHERE clause or similar to "find" the failed rows begs the question of why not just use that WHERE clause (or CASE clause) once to separate good/bad rows and skip using the DQ checks altogether i.e. MLVs are useful, but potentially the DQ checks are self-defeating and should not be used ?. 

It was very much the case that when I first read the MLV documentation my _first _question was how to get the rows failed by the DQ checks, in order for me to understand how I could use MLVs in production.  Everything else about the feature made sense to me, but not the DQ checks.

Hope some of this makes sense, and certainly happy for any clarifications or insights on how I might have the wrong perspective here.
 
cheers