Forum Discussion
RunMultiple DAG and tricks
Hi DennesTorres
The internal team replied as follows:
Just a suggestion, but seems like it's simply a problem with json->string->json conversion. Perhaps try json.dumps at notebook exit to safely convert the json to string and then json.loads once you've combined them together? Also remember the new soft limit for runMultiple is by default 50 so may be better to convert to python multithreading but note this will only run on the driver node so use single node pool
Hope this helps. Please let me know if you have any further questions.Hi,
Thank you. Yes, I found a similar solution.
The problem was the load. Making json.load before returning the data converts string -> dictionary . When returning the dictionary in the DAG, the conversion from dictionary -> string doesn't result in the same string.
The solution was to not use JSON load inside the parallel execution. I returned the original string, retrieved outside and used the json.load outside the parallel exeuction. It work perfectly.
Is the same as the recommendation.
About the runMultiple limit, I'm aware. About python multithreading, it's something I need to explore.Kind Regards,
Dennes
Hi DennesTorres
Thanks for using Fabric Community.
At this time, we are reaching out to the internal team to get some help on this. We will update you once we hear back from them.
Thanks
Hi DennesTorres
The internal team replied as follows:
Just a suggestion, but seems like it's simply a problem with json->string->json conversion. Perhaps try json.dumps at notebook exit to safely convert the json to string and then json.loads once you've combined them together? Also remember the new soft limit for runMultiple is by default 50 so may be better to convert to python multithreading but note this will only run on the driver node so use single node pool
Hope this helps. Please let me know if you have any further questions.
Hi,
Thank you. Yes, I found a similar solution.
The problem was the load. Making json.load before returning the data converts string -> dictionary . When returning the dictionary in the DAG, the conversion from dictionary -> string doesn't result in the same string.
The solution was to not use JSON load inside the parallel execution. I returned the original string, retrieved outside and used the json.load outside the parallel exeuction. It work perfectly.
Is the same as the recommendation.
About the runMultiple limit, I'm aware. About python multithreading, it's something I need to explore.Kind Regards,
Dennes