topic Re: copy activy runs infinitely in Pipelines

copy activy runs infinitely

Master — Wed, 28 Aug 2024 15:56:22 GMT

I have a copy activity to convert a json file to a parquet file that sometimes runs infinitely (and ends at the configured maximum timeout) but normally works fine. Does anyone have an idea what can cause this phenomenon?

Re: copy activy runs infinitely

Anonymous — Thu, 29 Aug 2024 01:44:11 GMT

Hi Master ,

The reasons why the replication activity may run indefinitely I believe are these:

1. if the JSON file is very large or contains complex nested structures, it may significantly increase processing time.

2. if your data source or target is experiencing network issues, this may cause delays or timeouts. This is more noticeable when you are using cloud storage or remote servers.

3. Insufficient resources (CPU, RAM) on the computer running the replication activity may cause the performance to not be able to keep up at all.

So I think you can try to run the replication activity with a smaller or simpler JSON file. Check the resource usage during the replication activity to see if there are any spikes or bottlenecks.

Translated with DeepL.com (free version)

Of course if you JSON has nested arrays, the solution is to use Flatten conversion in the data stream. You can check out this documentation for more information: API (JSON) to Parquet via DataFactory - Microsoft Q&A

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

Re: copy activy runs infinitely

Master — Thu, 29 Aug 2024 13:33:52 GMT

I simply use a copy activity in the Microsoft Fabric pipeline to do this conversion

Re: copy activy runs infinitely

Master — Tue, 03 Sep 2024 12:28:28 GMT

For your information, my solution to this problem was simply to replace my copy activity with a small python notebook to do this conversion. In python, it never bugs.