Is it possiable to event trigger pipelines on a hybrid on premise&cloud architecture

[WillemLiang]

@v-nikhilan-msft I've saw a similar issue from here: https://community.fabric.microsoft.com/t5/Data-Pipelines/Is-it-possible-to-do-an-event-based-trigger..., since it has been marked solved and not suit for my case, so I decide to create a new one as follows:

We're migrating some workloads to Azure, but due to data outbound and security issues, we should building a on premise(Hadoop) and cloud (ms-fabric) architecture. We've built Hive data warehouse and it running well for over years, Some data we've devolped on hadoop and I'd like to push it into azure blob storage and invoke pipeline by an event trigger so that it will be no needs to imprement it on fabric as well. 

So basicly I have devoloped Hive tables and them would be pushed to azure blob by hadoop `dict_cp`  and I expecting Fabric pipelines triggered once the blob created. My team is skilled in Hadoop and comfortable with spark, python, SQL and java, but new to Azure and MS Fabric.

The problems are:

1. Feature not ready: event trigger is not supportd in Fabric version ADF

2. Trigger by folder match: If data push task split into many maps, then destination files in blob storage will have same file count, then pipeline will be triggered for many times. Apparently it can be solved by setting the map number to 1, but it not elegant anyway.

3. Trigger once and tasks queue automation in a many source pipeline: If a pipeline dependencis many blob source data, I wish the pipeline could be running in a single instance and all the storage event triggered then bring it's folderPath to tell the downstream activities the needed data is ready. When a folder created, then the pipeline invoke, the downstream jobs push into a "task queue" and wating to run till all it's dependencis are ready, and when new data comes, no more pipeline runs and only pop the dependencis in the downstream task queue list. For the downstream task,It will be run when empty it's dependencis list.

Either Fabric and ADF lacks those features and have too many limitations for my team, for this specific and not ideal data fabric scenario, are there any solutions?

[v-nikhilan-msft]

Hi @WillemLiang 
Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.
Thanks 

[v-nikhilan-msft]

Hi @WillemLiang 
The internal team has replied as follows:

Event triggers for pipelines will be soon released in Fabric. Its on roadmap. We dont have any ETA yet. I will keep you posted regarding the updates.

Hope this helps. Please let me know if you have any further questions.

[WillemLiang · Accepted solution]

Thanks for the updates. We're moving to dbt solution to see if there are any differences, but we still prefer to evolve with fabric and looking forward to it becoming the primitive feature, please keep us posted.

[v-nikhilan-msft]

Hi @WillemLiang 
Thanks for your feedback. I will keep you posted regarding the updates. Please continue using Fabric Community for any help regarding your queries.

[v-nikhilan-msft]

Hi @WillemLiang 
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks

[v-nikhilan-msft]

Hi @WillemLiang 
We haven’t heard from you on the last response and was just checking back to see if your query got resolved. Otherwise, will respond back with the more details and we will try to help.
Thanks