Question: What does ETL always need in production?
Answer: Messaging on failures
Q: What is the best channel today?
A: Teams (and Email because you can make email rules easier)
Q: Can you do this in Fabric as of 2024/07?
A: Yes, messaging to Teams is in preview state.
What doesn't work:
- Data pipelines always use AND when combining several inputs. This means that you must use OR box to combine failures together before you report the failure. As Data Pipeline doesn't have an OR box, you must create failure reporting for every action separately.
- You can't copy-paste Teams-messaging or their content, so you have to make all of them manually one by one.
- I am not sure if can even copy paste functions, the UI is difficult.
- You can't tell what environment you are running the Data Pipeline: Dev, Test or Prod
Now this idea suggesting is on the last one:
- You can now drag Workspace ID to the Teams Message. This gets renamed as DataFactory for some reason, but don't get confused with that.
- Problem is that this ID is a cryptic key such as 8466c1bc-e477-415e-8b77-a714cd012345, so you need to add separately a decrypt table so people will know what environment was this run on.
So Data Pipeline should have "Workspace" or "Workspace name" dynamic content to choose from.
4 Comments
I don't understand why this is not implemented initially.
Duplicate ideas:
https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=a6df1149-1495-ee11-a81c-000d3a00bef9
https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=b3ccfa82-eef7-4d52-ba47-79aed6d96466
https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=1eab5cdc-776a-430f-98e7-b64300f50576
https://ideas.fabric.microsoft.com/ideas/idea/?ideaid=2b777168-fc47-ef11-b4ad-000d3a0f9a18
I'd suggest looking into Activators to stream pipeline failures to alerts. You can get pretty granular with rules, and it supports both emails and teams messages as actions. I think these are relatively new, so might not have been available at the time this was submitted.
Here's the MS link for the documentation:
https://learn.microsoft.com/en-us/fabric/real-time-intelligence/data-activator/activator-introduction
The activators will give you direct access to things like the workspace name, with nothing more than a click.
But, just so you are aware, it is also possible to use sempy.fabric to programmatically identify workspace names and objects. You shouldn't store a static list which would need to be maintained.
https://learn.microsoft.com/en-us/python/api/semantic-link-sempy/sempy.fabric?view=semantic-link-python
- Status added:Needs Votes
- SJCuthbertson
Advocate I
I think https://community.fabric.microsoft.com/t5/Fabric-Ideas/Need-to-grab-the-Workspace-Name-from-the-System-Variables/idc-p/4645389 is an exact duplicate - that's one of the ones listed in a separate comment last year by Niels, but the URLs have changed it seems. I think the other URLs provided by Niels may not be exact duplicates, and are more related to the emails that Power BI sends when a semantic model refresh fails. Similar motivation (humans can't understand GUIDs) but different context.
