data pipeline
5 Topics
How to execute multiple dataflows(Dataflow Gen2 CI/CD Enabled) dynamically in Fabric?
Hi Community, I have more than 40 dataflows that I need to refresh/execute. Right now, I have to set them up one by one, which is time-consuming. Is there a way to dynamically execute all my dataflows together (for example, through a script, API, pipeline, or any other approach), instead of triggering them individually? Any guidance or best practices on how to manage and schedule multiple dataflows efficiently would be very helpful.Solved4KViews0likes6Comments
HTTP GET request not working in Copy Data activity, although same URL works in Postman
I am currently trying to submit a HTTP GET request in a Copy Data activity for a Data Pipeline in Data Factory. I have tried three different URLs from the three websites I am trying to copy data from, and am met with the same response each time: "Bad data is found at line 1 in source ... You can ignore bad data by setting BadDataFound to null. IReader state: ColumnCount: 27345 CurrentIndex: -1 HeaderRecord: IParser state: ByteCount: 0 CharCount: 1077997 Row: 1 RawRow: 1 Count: 27345 RawRecord: Hidden because ExceptionMessagesContainRawData is false." I cannot edit the source data/schema as its coming from a read-only URL. There is no setting "BadDataFound" within the Copy Data activity - rendering that option completely useless. The "Fault tolerance" setting within the Copy Data activity is set to "Skip incompatible rows" and that has not helped. This API GET request works completely fine within Postman. Wondering what the issue is, if Data Factory doesn't actually support this or if I'm missing something completely.Solved1.1KViews0likes1Comment
