Hello!
TLDR: is there a way of setting up an alert for when an eventstream fails to ingest data? If not, this is an Idea for Fabric.
I know that you can monitor the eventstream incoming and outcoming data, as well as checking the runtime logs. What I want is for instance an automatic alert when a runtime error is observed in the Runtime Log of the EventStream.
The reason why we need this:
Im working at a project were we use eventstreams to ingest incoming data. Several times we have experienced that the eventstream fails, so far it has been due to the following reasons:
- Deactivating the Fabrics user that was used to create the eventstream
- Downgrading the privelage of the user that created / OR EDITED the eventstream
- More specifically: we had created a set of eventstreams with a serviceaccount that has the highest privelages. However, minor details of the eventstream had been edited with our private accounts. Yesterday we lowered our private accounts to READ ONLY in the workspace, and this made the eventstreams fail, presumably since the last account that edited the evenstreams were these private accounts, and they do not have write access to the bronze layer anymore.
We can not accept the data in our bronze layer not being up to date. At the same time we do not have the capactiy to check up on all eventstreams every now and then to check that they work as intended.
Thanks in advance!
3 Comments
- nishalitNew Member
Thanks for this great idea. Product team takes this into our backlog and continue collecting more vote for this idea.
- fbcideas_migusrNew MemberStatus added:Needs Votes
- joezhang-es
Microsoft Employee
thanks to share your scenario in detail which is very helpful for our PG to understand your pain points the solution will be - onboard event stream to fabric workspace monitoring then all the runtime logs and metrics you are monitoring can be flowing into event house Kusto table then you can build real time dashboard or set alert with data activator aganist this
Recent ideas
Allow NotebookUtils getSecret() to authenticate with Workspace Identity
Current behavior In Microsoft Fabric, notebookutils.credentials.getSecret() authenticates against Azure Key Vault using the identity of the user who executes the notebook. This behavior appli...tmihara1 hour agoNew MemberNew16Views1like1CommentAllow us to rename fabric data agents published to m365
If we use deployment pipelines to promote fabric data agents between dev, test/UAT, and prod fabric workspaces, we need to keep the name of the fabric agent the same in each workspace. If we want to ...PeterDaniels8 hours agoAdvocate IIINew184Views0likes2CommentsMake workspace and item session persistence optional
Description The new persistent session behavior in Microsoft Fabric should be optional rather than forced. Currently, Fabric remembers the workspaces and items that were open in my previous session...TeemuMultanen9 hours agoAdvocate INew299Views47likes2Comments