Curl Multi

[dawinaj]

Hello,
are there any plans of implementing a parallelization with curl multi handles? I am especially interested in parallel upload (I have 6 directories with ~800 files each so doing it file-by-file takes a reaaaaaally long time).

I might even be interested to help implement that, however I don't have much experience with creating libraries, especially cross-platform...

Also, I think it would be nice if any action that failed (upload, download, etc) was automatically redone.

[embeddedmz]

Hi herhor69,

It is not intended to add such features to the class. You will have to code your own solution for these problems as there may be several ways/constraints to implement such features (e.g. it's like the C++ standard library std::string class, it lacks many practical methods like to_upper() because a std::string can contain UTF-8 strings and with some languages that do not have the concept of capital letters, how the method should behave ?).

A retry mechanism is easy to implement (a while loop, a counter variable, a maximum number of attempts and the return value of the upload method) and to parallelize the uploads I see it like this: first you create lists of upload items/jobs in a single thread (a data structure that contains the path of the local file to be uploaded and the future path of this file on the FTP server) and each of these lists will be processed by a dedicated thread that will have a dedicated CFTPClient object. If an upload fails you can use a retry mechanism to upload it again, print the error somewhere, etc...

Pay attention to thread management (stop threads with a "cancellation token" (an atomic bool), data race, wait for threads to finish correctly with a join method/function...). Personally, when it's possible, I prefer to use Qt threads and the event loop mechanism to manage threads (message passing/actor pattern) because we have a more deterministic behavior. In C# I have already made something that looks like Java's Executor class to execute many lambdas in a single thread. I don't like thread pools or in other terms creating threads dynamically.

Best regards.