Unix and Linux Stack Exchange
Unix and Linux Stack Exchange

Questions tagged [parallelism]

Performing tasks in parallel, to make use of multiple processors

Parallelism, or parallelization, or parallel computing, means making use of multiple processors to complete a task more quickly than on a single processor.

For using multiple threads when the presence of more than one processor is irrelevant, use or instead. For parallelism over multiple machines, and not just over multiple processors in the same machine, use in addition to or instead of this tag.

is a common tool for parallel execution.

Useful links

273 questions
78
votes
6 answers

How to determine the maximum number to pass to make -j option?

I want to compile as fast as possible. Go figure. And would like to automate the choice of the number following the -j option. How can I programmatically choose that value, e.g. in a shell script? Is the output of nproc equivalent to the number…
tarabyte
  • 4,166
  • 10
  • 36
  • 48
78
votes
3 answers

top command on multi core processor

I am using freescale IMX6 quad processor. I want to know if the top command lists the CPU usage of all 4 cores or of a single core. I am seeing an application's CPU usage being the same with 4 cores and with a single core. I was guessing the CPU…
user3818847
  • 1,131
  • 2
  • 8
  • 7
63
votes
16 answers

Automatically run commands over SSH on many servers

There is a list of IP addresses in a .txt file, ex.: 1.1.1.1 2.2.2.2 3.3.3.3 Behind every IP address there is a server, and on every server there is an sshd running on port 22. Not every server is in the known_hosts list (on my PC, Ubuntu 10.04…
LanceBaynes
  • 39,295
  • 97
  • 250
  • 349
58
votes
15 answers

How do I run the same linux command in more than one tab/shell simultaneously?

Is there any tool/command in Linux that I can use to run a command in more than one tab simultaneously? I want to run the same command: ./myprog argument1 argument2 simultaneously in more than one shell to check if the mutexes are working fine in a…
Arpith
  • 1,091
  • 3
  • 14
  • 23
52
votes
5 answers

How to start multi-threaded grep in terminal?

I have a folder which has 250+ files of 2 GB each. I need to search for a string/pattern in those files and output the result in an output file. I know I can run the following command, but it is too slow!! grep mypattern * > output I want to speed…
Abhishek
  • 629
  • 1
  • 6
  • 10
39
votes
2 answers

How do I distribute a large download over multiple computers?

I need to download a large file (1GB). I also have access to multiple computers running Linux, but each is limited to a 50kB/s download speed by an admin policy. How do I distribute downloading this file on several computers and merge them after all…
B Faley
  • 4,213
  • 11
  • 37
  • 48
30
votes
7 answers

Simultaneously calculate multiple digests (md5, sha256)?

Under the assumption that disk I/O and free RAM is a bottleneck (while CPU time is not the limitation), does a tool exist that can calculate multiple message digests at once? I am particularly interested in calculating the MD-5 and SHA-256 digests…
Lekensteyn
  • 20,173
  • 18
  • 71
  • 111
25
votes
4 answers

Four tasks in parallel... how do I do that?

I have a bunch of PNG images on a directory. I have an application called pngout that I run to compress these images. This application is called by a script I did. The problem is that this script does one at a time, something like…
Duck
  • 4,434
  • 19
  • 51
  • 64
24
votes
9 answers

Calling multiple bash scripts and running them in parallel, not in sequence

Suppose that I have three (or more) bash scripts: script1.sh, script2.sh, and script3.sh. I would like to call all three of these scripts and run them in parallel. One way to do this is to just execute the following commands: nohup bash script1.sh…
Andrew
  • 16,315
  • 34
  • 73
  • 77
23
votes
2 answers

Executing piped commands in parallel

Consider the following scenario. I have two programs A and B. Program A outputs to stdout lines of strings while program B process lines from stdin. The way to use these two programs is of course: foo@bar:~$ A | B Now I've noticed that this eats…
Jernej
  • 367
  • 1
  • 2
  • 8
22
votes
2 answers

How to stop xargs from badly merging output from multiple processes?

I'm using xargs with the option --max-args=0 (alternatively -P 0). However, the output of the processes is merged into the stdout stream without regard for proper line separation. So I'll often end up with lines such…
Christoph Wurm
  • 5,678
  • 5
  • 25
  • 28
18
votes
4 answers

Are basic POSIX utilities parallelized?

In a common Linux distribution, do utilities like rm, mv, ls, grep, wc, etc. run in parallel on their arguments? In other words, if I grep a huge file on a 32-threaded CPU, will it go faster than on dual-core CPU?
homocomputeris
  • 391
  • 2
  • 13
18
votes
6 answers

using parallel to process unique input files to unique output files

I have a shell scripting problem where I'm given a directory full of input files (each file containing many input lines), and I need to process them individually, redirecting each of their outputs to a unique file (aka, file_1.input needs to be…
J Jones
  • 325
  • 1
  • 3
  • 8
17
votes
2 answers

How can I get the index of the xargs "parallel processor"?

Suppose I have two resources, named 0 and 1, that can only be accessed exclusively. Is there any way to recover the "index" of the "parallel processor" that xargs launches in order to use it as a free mutual exclusion service? E.g., consider the…
VF1
  • 403
  • 4
  • 8
17
votes
6 answers

Is there parallel wget? Something like fping but only for downloading?

I've found only puf (Parallel URL fetcher) but I couldn't get it to read urls from a file; something like puf < urls.txt does not work either. The operating system installed on the server is Ubuntu.
Moonwalker
  • 313
  • 1
  • 2
  • 5
1
2 3
…
18 19