27

I start off in empty directory.

$ touch aFile
$ ls
aFile

Then I ls two arguments, one of which isn't in this directory. I redirect both output streams to a file named output. I use >> in order to avoid writing simultaneously.

$ ls aFile not_exist >>output 2>>output
$ cat output
ls: cannot access 'not_exist': No such file or directory
aFile

Which seems to work. Are there any dangers to this approach?

exit_status
  • 327
  • 4
  • 10
  • 8
    That was a fast down vote. It took like five seconds. Can you tell me how is that you can assess the worthiness of my question so quickly? And better yet, what is wrong with it so that I can improve it? – exit_status May 19 '19 at 10:35
  • Why don't you use the more standard `ls aFile not_exist &>>output` here? (Note, I am assuming you are using *bash*.) – FedKad May 19 '19 at 10:47
  • 6
    Because that doesn't help me understand what I'm asking about. I know how to redirect these streams to the same file, portably even. What I want to know is if there's anything wrong with the what I suggested in the question. @FedonKadifeli – exit_status May 19 '19 at 10:52
  • I guess both forms (yours and mine) are equivalent, but it is difficult to speculate about nuances, unless you inspect the source code of the shell interpreter (_bash_). – FedKad May 19 '19 at 10:58
  • 1
    @FedonKadifeli `&>>` is NOT standard. It's a DEPRECATED, ambiguous syntax which works differently in different shells. I wonder where you guys get your stuff from. –  May 19 '19 at 11:32
  • @exit_status I see no problem with that, but it's pretty pointless to open the same file twice (and waste another open file description state structure in the kernel) instead of opening it once and dup-ing the fd. What's the problem with `cmd ... >>logfile 2>&1`? –  May 19 '19 at 11:37
  • @Uncle Billy. I am getting this syntax directly from the _bash_ man page: http://manpages.ubuntu.com/manpages/disco/man1/bash.1.html (Section: **Appending Standard Output and Standard Error**). I specifically mentioned _bash_. What is your source about being it DEPRECATED??? – FedKad May 19 '19 at 11:42
  • 4
    Bash is not a *standard*. The POSIX standard mandates that `ls &>>foo ...` should be parsed as two comands `ls &` and `>>foo ...`, and this is the way other shells like the `/bin/sh` from Ubuntu are parsing it. For it being deprecated, you can look [here](https://wiki-dev.bash-hackers.org/scripting/obsolete) -- though I don't pretend that's any kind of authority. You may ask the `bash` maintainers if they consider using that a good idea, though. –  May 19 '19 at 11:50

3 Answers3

22

What happens when you do

some_command >>file 2>>file

is that file will be opened for appending twice. This is safe to do on a POSIX filesystem. Any write that happens to the file when it's opened for appending will occur at the end of the file, regardless of whether the data comes over the standard output stream or the standard error stream.

This relies on support for atomic append write operations in the underlying filesystem. Some filesystems, such as NFS, does not support atomic append. See e.g. the question "Is file append atomic in UNIX? " on StackOverflow.

Using

some_command >>file 2>&1

would work even on NFS though.

However, using

some_command >file 2>file

is not safe, as the shell will truncate the output file (twice) and any writing that happens on either stream will overwrite the data already written by the other stream.

Example:

$ { echo hello; echo abc >&2; } >file 2>file
$ cat file
abc
o

The hello string is written first (with a terminating newline), and then the string abc followed by a newline is written from standard error, overwriting the hell. The result is the string abc with a newline, followed by what's left of the first echo output, an o and a newline.

Swapping the two echo around wound produce only hello in the output file as that string is written last and is longer than the abc string. The order in which the redirections occur does not matter.

It would be better and safer to use the more idiomatic

some_command >file 2>&1
Kusalananda
  • 320,670
  • 36
  • 633
  • 936
  • 1
    While that's true of modern shells, that was not the case in the Bourne or Thomson shell (where `>>` comes from), where `>>` would open for writing and seek to the end (I suppose because O_APPEND wasn't invented yet back then). Even on Solaris 10, `/bin/sh -c '(echo a; echo b >&2) >> file 2>> file; cat file'` outputs `b`. – Stéphane Chazelas May 19 '19 at 18:47
  • @StéphaneChazelas Is that an issue with Solaris 10's implementation of `sh`, or with its filesystem? – Kusalananda May 19 '19 at 18:49
  • 1
    That's what `>>` was originally doing, it was not opening with O_APPEND, it was opening without and seeking to the end. It's not that much an issue, it's what it was doing and was documented to do. – Stéphane Chazelas May 19 '19 at 19:09
22

No, it's not just as safe as the standard >>bar 2>&1.

When you're writing

foo >>bar 2>>bar

you're opening the bar file twice with O_APPEND, creating two completely independent file objects[1], each with its own state (pointer, open modes, etc).

This is very much unlike 2>&1 which is just calling the dup(2) system call, and makes the stderr and stdout interchangeable aliases for the same file object.

Now, there's a problem with that:

O_APPEND may lead to corrupted files on NFS filesystems if more than one process appends data to a file at once. This is because NFS does not support appending to a file, so the client kernel has to simulate it, which can't be done without a race condition.

You usually can count on the probability of the file like bar in foo >>bar 2>&1 being written to at the same time from two separate places being quite low. But by your >>bar 2>>bar you just increased it by a dozen orders of magnitude, without any reason.

[1] "Open File Descriptions" in POSIX lingo.

  • 3
    Formally, for append-mode files, it **is safe**. The cited issue is a **bug** in NFS that makes it unsuitable (non-POSIX-conforming) as a filesystem. For the non-append-mode case, though, it's never safe. – R.. GitHub STOP HELPING ICE May 20 '19 at 04:14
  • 1
    That's immaterial. The OP's double-append is **not** safe to use (in addition to being completely pointless). And `O_APPEND` is kind of a botch anyway -- pretty onerous to implement correctly. –  May 20 '19 at 11:17
  • I believe the NFS race condition is only between different clients. The client OS should coordinate all the writes between its processes. – Barmar May 20 '19 at 17:36
  • @Barmar that would be true if the client OS only cared about its own view of a nfs file. But when writing to nfs file opened with `O_APPEND`, the client will first retrieve the "real" size of the file from the server ("revalidate" the inode) and then do the seek+write+cached inode update, and only the last part is done under locks, which means that the first part could still retrieve a stale size from the server and override the correct one from the local/cached inode. Same problem with `lseek(SEEK_END)`. –  May 20 '19 at 23:46
  • I still don't see how that could cause race conditions between two streams on the same client. Both streams should refer to the same local cached inode. – Barmar May 20 '19 at 23:56
  • When the file is opened with `O_APPEND`, the client OS will retrieve the file size from the *server*, not from its cached inode. By the time a process A gets the answer from server and updates the cached inode, another process B on the same machine may write to it, an the process A will overwrite the correct inode info with the stale one retrieved from the server. –  May 21 '19 at 00:03
0

It depends what you want to achieve. It's up to you to decide is it OK to have errors in the same file as the output. This is just saving text in a file with the functionality of the shell which let you redirect as you wish. There is no absolute yes or no.As everything in Linux it can be done in several ways , this is my way ls notExistingFile existingFile >> output 2>&1 To answer the question: In terms of the redirecting itself , yes its perfectly safe.

Angel
  • 310
  • 2
  • 10
  • There's more to it than what you're saying here. The same exercise with `>` instead of `>>` will overwrite some characters. So it's not just that the shell allows me to redirect, because when I redirect with `>`, the result is different. So there are nuances with `>`, are there any with `>>`? – exit_status May 19 '19 at 11:12
  • Yes it will be different . As i said it depends on your goal `>`- overwrite . `>>` - append – Angel May 19 '19 at 11:20