Skip to main content
cancel
Showing results for 
Search instead for 
Did you mean: 

Get certified as a Fabric Data Engineer: Check your eligibility for a 50% exam voucher offer and join us for free live learning sessions to get prepared for Exam DP-700. Get started

Reply
Master
Helper I
Helper I

copy activy runs infinitely

I have a copy activity to convert a json file to a parquet file that sometimes runs infinitely (and ends at the configured maximum timeout) but normally works fine. Does anyone have an idea what can cause this phenomenon?

1 ACCEPTED SOLUTION
Master
Helper I
Helper I

For your information, my solution to this problem was simply to replace my copy activity with a small python notebook to do this conversion. In python, it never bugs.

View solution in original post

3 REPLIES 3
Master
Helper I
Helper I

For your information, my solution to this problem was simply to replace my copy activity with a small python notebook to do this conversion. In python, it never bugs.

v-yilong-msft
Community Support
Community Support

Hi @Master ,

The reasons why the replication activity may run indefinitely I believe are these:

1. if the JSON file is very large or contains complex nested structures, it may significantly increase processing time.

2. if your data source or target is experiencing network issues, this may cause delays or timeouts. This is more noticeable when you are using cloud storage or remote servers.

3. Insufficient resources (CPU, RAM) on the computer running the replication activity may cause the performance to not be able to keep up at all.

 

So I think you can try to run the replication activity with a smaller or simpler JSON file. Check the resource usage during the replication activity to see if there are any spikes or bottlenecks.

Translated with DeepL.com (free version)

 

vyilongmsft_0-1724895299147.png

Of course if you JSON has nested arrays, the solution is to use Flatten conversion in the data stream. You can check out this documentation for more information: API (JSON) to Parquet via DataFactory - Microsoft Q&A

 

 

 

Best Regards

Yilong Zhou

If this post helps, then please consider Accept it as the solution to help the other members find it more quickly.

I simply use a copy activity in the Microsoft Fabric pipeline to do this conversion

Helpful resources

Announcements
Feb2025 Sticker Challenge

Join our Community Sticker Challenge 2025

If you love stickers, then you will definitely want to check out our Community Sticker Challenge!

JanFabricDE_carousel

Fabric Monthly Update - January 2025

Explore the power of Python Notebooks in Fabric!

JanFabricDW_carousel

Fabric Monthly Update - January 2025

Unlock the latest Fabric Data Warehouse upgrades!