Forum Discussion

AJAJ's avatar
AJAJ
Helper IV
7 months ago
Solved

Copy Activity Dynamic Source Connection

Hi Currently my pipelines on dev and prod have the same source. So basically my ingestion pipeline + EDW pipeline run daily. Reads same source twice. By prod and then by Dev. In order to avoid doub...
data engineering
Data Pipeline
  • v-menakakota's avatar
    v-menakakota
    7 months ago

    Hi AJAJ ,

    I hope the above details help you fix the issue. If you still have any questions or need more help, feel free to reach out. We’re always here to support you .

     

     

    Best Regards, 
    Community Support Team

AJAJ's avatar
AJAJ
Helper IV
7 months ago

cengizhanarslan tagging you if you happen to have come across the above question.

 

Wondering if there is anyway to have 1 notebook copying data from different sources using foreach loop and pushing the data into different destinations or same destinations (refer to excel tab above which has source and destination). Some sources are ODBC, some sources are SQL server, some are SFTP folders. Just need to loop and copy based on the query. In future Col4 might be added to s1_tableA.

Importantly things should run in parallel across sources. sometimes run 10 parallel extraction within the same source.