6

I have been reading up on control operators, and I was wondering if there was a limit to how many commands you could line up with control operators, such as ||, && and ;.

In addition, is there a configuration file somewhere where this can be regulated?

PS: I am not entirely sure how to tag this.

Adam Jensen
  • 277
  • 2
  • 8

2 Answers2

7

There isn't really; as long your computer's memory can handle the queue, the shell should do its best. According to POSIX:

The shell shall read its input in terms of lines from a file, from a terminal in the case of an interactive shell, or from a string in the case of sh -c or system(). The input lines can be of unlimited length. These lines shall be parsed using two major modes: ordinary token recognition and processing of here-documents.

Basically all of those || && strung together amount to a single input line for the shell's parser, because it has to parse tokens for each command list before then evaluating and executing the list's constituent simple commands.

I once covered something like this here - and there are a lot of command examples there detailing how the parser works (or at least how I understand it works).

Community
  • 1
mikeserv
  • 57,448
  • 9
  • 113
  • 229
  • 2
    Awesome. Is it safe to state that the _only_ limitation is the device's memory? – Adam Jensen Aug 06 '14 at 01:45
  • but is using a lot of piping or any other control operators will be bad for performance – Nidal Aug 06 '14 at 01:46
  • 1
    @AdamJensen - far from it. The biggest limiter I usually run into is the shell. They attempt to do all kinds of weird things in order to deal with input of *unlimited length* - and it often limits it. Still, I do rather like the way `dash` handles here-documents - it uses anonymous pipes and can be very handy - and very fast. Shells usually exhaust stack or otherwise go out of band long before my 24gbs of RAM is consumed. – mikeserv Aug 06 '14 at 01:53
  • 1
    @Networker - to the contrary, generally pipes are what you want because - if the shell does it right - it seats ipc firmly in kernelspace. The shell just has to do the initial forks and opens and whatever then wait for sigchild. But you probably shouldn't jam them all into one line if it can be helped - and you can use outside files like FIFOs or whatever to link pipelines as you like. `nc` is pretty handy for that too. – mikeserv Aug 06 '14 at 01:59
  • 1
    @mikeserv, I asked this because I use piping a lot and when I read this sentence `If you're compairing longer pipelines of simpler tools with a single invocation of a more complex tool` from [Gilles](http://unix.stackexchange.com/users/885/gilles) answer to my [question](http://unix.stackexchange.com/questions/132313/what-are-the-exact-differences-between-awk-and-cut-with-grep) I thought it is bad for using a lot of piping, the same for [Avinash Raj](http://unix.stackexchange.com/users/56901/avinash-raj) answer – Nidal Aug 06 '14 at 02:10
  • 1
    @Networker - I've just read that answer and I get the impression he's saying the opposite. Or, he does not commit. He says *more specialized tools are faster* - as in the less complex and more focused - and he says *no rule of thumb* regarding *long pipelines* about whether to use specialized tools - small ones - or complicated tools, like `awk`. He definitely does make a good case about why it may be *hard* to use a lot of piping - as in the different commands talk different languages - but no case for or against performance or security. – mikeserv Aug 06 '14 at 02:30
0

I believe that you will find the limit you encounter is not the number of commands but the size of the sh command buffer. On current Linux systems I believe that the size is about 64K characters.

mdpc
  • 6,736
  • 3
  • 32
  • 46
  • That is the number of parsed arguments that can be passed....No, I am talking about the sh internal command line buffer length. – mdpc Aug 06 '14 at 02:22
  • Ahh...whatever the shell program is..sh bash csh tcsh...whatever. – mdpc Aug 06 '14 at 02:23
  • you'd need to look at the source code for the shell you are wondering about. Since this is Linux, it IS available and open for inspection. – mdpc Aug 06 '14 at 02:44
  • come on, man. can you clear that up in the answer? because this is Linux `sh` could be *any* shell. Where do you get 64k for *any* shell? – mikeserv Aug 06 '14 at 02:50
  • The maximum line length is not determined by the shell: [it is determined by the OS](http://stackoverflow.com/a/19355351/3030305): To determine the setting on a [Linux/UNIX/BSD system](http://www.cyberciti.biz/faq/argument-list-too-long-error-solution/), run `getconf ARG_MAX`. – John1024 Aug 06 '14 at 05:23
  • @John1024 - as mdpc said before, *(because I thought he was referring to same)* - *That is the number of parsed arguments that can be passed....* Those arguments are *passed* at process invocation time - and, for the most part, have very little to do with the shell because it generally only gets invoked the once like `exec ENV=/path/to/env/file sh`. So that environment file can have lines far longer than ARG_MAX because the shell will open() and read it in on a file descriptor. ARG_MAX probably only applies like `sh -c "$@"` or something. – mikeserv Aug 06 '14 at 07:11
  • @mikeserv Try it. For example, either on the command line or as part of a shell script, try the line `touch aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa{1..100000}`. When run, the shell will try to expand it and it will return the error message: `/bin/touch: Argument list too long`. – John1024 Aug 06 '14 at 07:38
  • @John1024 `touch` invokes a new process - it apparently here calls `/bin/touch`. So the shell can pass to touch only so many arguments as the kernel will allow on a single cmdline. So it dumps and quits. This has almost nothing at all to do with the length of the line that the shell read in before making that invocation - just that you see `/bin/touch` there is a clear indication that the shell successfully processed and evaluated the entire line before trying - else you would see `/bin/bash` or something in that error output. – mikeserv Aug 06 '14 at 08:20