The ultimate Microsoft Fabric, Power BI, Azure AI, and SQL learning event! Join us in Stockholm, Sweden from September 24-27, 2024.
2-for-1 sale on June 20 only!
Find everything you need to get certified on Fabric—skills challenges, live sessions, exam prep, role guidance, and more. Get started
@v-nikhilan-msft I've saw a similar issue from here: https://community.fabric.microsoft.com/t5/Data-Pipelines/Is-it-possible-to-do-an-event-based-trigger..., since it has been marked solved and not suit for my case, so I decide to create a new one as follows:
We're migrating some workloads to Azure, but due to data outbound and security issues, we should building a on premise(Hadoop) and cloud (ms-fabric) architecture. We've built Hive data warehouse and it running well for over years, Some data we've devolped on hadoop and I'd like to push it into azure blob storage and invoke pipeline by an event trigger so that it will be no needs to imprement it on fabric as well.
So basicly I have devoloped Hive tables and them would be pushed to azure blob by hadoop `dict_cp` and I expecting Fabric pipelines triggered once the blob created. My team is skilled in Hadoop and comfortable with spark, python, SQL and java, but new to Azure and MS Fabric.
The problems are:
1. Feature not ready: event trigger is not supportd in Fabric version ADF
2. Trigger by folder match: If data push task split into many maps, then destination files in blob storage will have same file count, then pipeline will be triggered for many times. Apparently it can be solved by setting the map number to 1, but it not elegant anyway.
3. Trigger once and tasks queue automation in a many source pipeline: If a pipeline dependencis many blob source data, I wish the pipeline could be running in a single instance and all the storage event triggered then bring it's folderPath to tell the downstream activities the needed data is ready. When a folder created, then the pipeline invoke, the downstream jobs push into a "task queue" and wating to run till all it's dependencis are ready, and when new data comes, no more pipeline runs and only pop the dependencis in the downstream task queue list. For the downstream task,It will be run when empty it's dependencis list.
Either Fabric and ADF lacks those features and have too many limitations for my team, for this specific and not ideal data fabric scenario, are there any solutions?
Solved! Go to Solution.
Thanks for the updates. We're moving to dbt solution to see if there are any differences, but we still prefer to evolve with fabric and looking forward to it becoming the primitive feature, please keep us posted.
Hi @WillemLiang
Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.
Thanks
Hi @WillemLiang
The internal team has replied as follows:
Event triggers for pipelines will be soon released in Fabric. Its on roadmap. We dont have any ETA yet. I will keep you posted regarding the updates.
Hope this helps. Please let me know if you have any further questions.
Thanks for the updates. We're moving to dbt solution to see if there are any differences, but we still prefer to evolve with fabric and looking forward to it becoming the primitive feature, please keep us posted.
Hi @WillemLiang
Thanks for your feedback. I will keep you posted regarding the updates. Please continue using Fabric Community for any help regarding your queries.
Hi @WillemLiang
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks
Hi @WillemLiang
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks
Join the community in Stockholm for expert Microsoft Fabric learning including a very exciting keynote from Arun Ulag, Corporate Vice President, Azure Data.
Ask questions in Eventhouse and KQL, Eventstream, and Reflex.
User | Count |
---|---|
15 | |
3 | |
2 | |
1 | |
1 |