gNU parallel
parallel is a command-line driven utility for Linux or other Unix-like operating systems which allows the user to execute "for ~ do ~ done" like shell loops in parallel. parallel is a free software. It is available[1] under the terms of GPLv3.
Usage
The most common usage is to replace the shell loop, for example
(for x in `cat list` ; do
do_something $x
done) | process_output
to the form of
cat list | parallel do_something | process_output
where the file list contains arguments for do_something.
Scripts using parallel is often easier to read than scripts using pexec.
The program parallel features also
- grouping of output and error so it does not not run together;
- dealing nicely with file names containing special characters such as space, single quote, double quote, ampersand;
By default, parallel tries to detect the number of CPUs and uses all of them.