13

It's well known that redirecting standard output and error to the same file with cmd >out_err.txt 2>out_err.txt can lead to loss of data, as per the example below:

work:/tmp$ touch file.txt
work:/tmp$ ls another_file.txt
ls: cannot access 'another_file.txt': No such file or directory

The above is the setup code for the example. An empty file file.txt exists and another_file.txt is not a thing. In the code below, I naively redirect to out_err.txt both input and output os listing these files.

work:/tmp$ ls file.txt another_file.txt >out_err.txt 2>out_err.txt
work:/tmp$ cat out_err.txt 
file.txt
t access 'another_file.txt': No such file or directory

And we see that we lost a few characters in the error stream. However, using >> works in the sense that replicating the example would yield keep the whole output and the whole error.

Why and how does cmd >>out_err.txt 2>>out_err.txt work?

Sweet Shell O'Mine
  • 161
  • 8
  • I'm not sure I understand the question. When you use > any new output will overwrite the previous output as the filename is the same. >> will not overwrite but new output will always be added after the last line of the file; i.e. file is appended. Can you please clarify, maybe even with examples, what exactly it is you're after? – Peregrino69 Feb 06 '23 at 14:05
  • @Peregrino69 I added more detail. – Sweet Shell O'Mine Feb 06 '23 at 14:21
  • ... and I again learned something new from @ilkkachu :-) – Peregrino69 Feb 06 '23 at 14:42
  • 3
    "Why and how does `cmd >>out_err.txt 2>>out_err.txt` work?" – Somewhat related: [*Is redirection with `>>` equivalent to `>` when target file doesn't yet exist?*](https://superuser.com/q/1342489/432690) – Kamil Maciorowski Feb 06 '23 at 20:32

2 Answers2

23

Not sure it's that well known, but it happens because done like that, the two file handles are completely separate, and have independent read/write positions. Hence they can overwrite each other. (They correspond to two distinct open file descriptions, to use the technical term, which is sadly somewhat easy to confuse with the term "file descriptor".)

This only happens with foo > out.txt 2>out.txt, not with foo > out.txt 2>&1, since the latter copies the file descriptor (referring to the same open file description).

When appending, all writes go the to end of the file, as it is during the moment of the write. This is handled by the OS, atomically, so that there's no way for even another process to get in the middle. Hence, the issue from independent read/write positions is defused. (Except it might not work over NFS, that's a filesystem restriction.)

In your example, the error message ls: cannot access... is written first, at the start of the file. The write position of the stderr fd is now at the end of the file. Then the regular output of file.txt<newline> is also written, but the write position of the stdout fd is still at the start, so those 9 bytes overwrite part of the error message.

With an appending fd, that second write would go to end, regardless of anything.

ilkkachu
  • 133,243
  • 15
  • 236
  • 397
  • Thank you, this is helpful. Can you elaborate on how the reading and writing positions work? In my ignorant eyes, it is conceivable that while appending writes to the end of the file, both streams may be being writting portions of data to the end of the file one at a time. For example, in the example I now added to the question, stdout may write `fi` to the end of the file, then stderr may write `ls:` to the end of the file (resulting in the file starting with `fils:`) and so on and so forth. – Sweet Shell O'Mine Feb 06 '23 at 14:27
  • @SweetShellO'Mine, the output streams don't do anything them, it's the process making the system calls that causes the output. If `ls` calls `write(1, "foo", 3)`, those three bytes get written to stdout as one unit. If there's only one process, there's no question of ordering. If there was more than one, scheduling might after how their system calls are interleaved, but the data written in one call should still go as one atomic unit. – ilkkachu Feb 06 '23 at 20:20
  • (Technically, `write()` _can_ return after writing just some but not all of the given data, but in practice I'd be surprised if that happens when writing to a regular file. If it does happen, it's the job of the program to try again with the rest of the data, and anyway, with just one process involved, there still wouldn't be interleaving. Unless the program goes out of its way to do that on purpose.) – ilkkachu Feb 06 '23 at 20:23
  • Thanks, ikkachu. Is there only one process when using `>>` and more than one when using `>` in my example? – Sweet Shell O'Mine Feb 07 '23 at 00:56
  • I'm fairly sure that write to files are guaranteed atomic up-to the kernel buffer size (8K) writes to ttys are diferent, I'm not sure about writes to sockets, but I think pipes are also atomic to 8k. – Jasen Feb 07 '23 at 02:41
  • 2
    more info about file position can be found in the lseek(2) man page – Jasen Feb 07 '23 at 02:43
  • 2
    Minor nitpick: not all writes are atomic, the kernel only guarantees atomicity up to a certain size (I forget which constant it is off the top of my head). For many writes it won’t matter, but for large writes you will still see races – gntskn Feb 07 '23 at 05:16
  • @gntskn, it would be interesting to know how existing systems work in practice. The specification of write() goes to some lengths in describing how writes to a pipe or FIFO are atomic up to a certain length. That reads like an exception, but it doesn't really say there what the underlying rule is: if it's supposed to be not atomic at all for non-pipes, or atomic all the way. – ilkkachu Feb 07 '23 at 05:48
  • Section [2.9.7 Thread Interactions with Regular File Operations](https://pubs.opengroup.org/onlinepubs/9699919799/functions/V2_chap02.html#tag_15_09) however says "All of the following functions shall be atomic with respect to each other in the effects specified in POSIX.1-2017 when they operate on regular files or symbolic links: ... write(), writev()" so there appears to be an intent of atomicity, to some length? Though then again I found some article saying that's not the case for existing systems wrt. simultaneous writes and reads. Or maybe it only means threads within a single process? – ilkkachu Feb 07 '23 at 05:49
  • Also I haven't really seen a hard specification, the Linux man page doesn't seem to say, but I haven't looked further. If you have a link, I'd be happy to see one. Anyway, I do suspect what you're saying is essentially right, that writes up to some sensible length are atomic. If they weren't, anything appending to some sort of a log file from two processes would be effed. – ilkkachu Feb 07 '23 at 05:51
  • @SweetShellO'Mine, a process is a running instance of a program, you have one there, that `ls`. The file descriptors / output streams / whatever aren't processes, they don't have code that runs. – ilkkachu Feb 07 '23 at 06:18
  • Beside kernel atomicity of `write()` there's also the issue of stdio buffering. Unless writing to a terminal, `stdout` is fully buffered while `stderr` is line-buffered. – Barmar Feb 07 '23 at 15:30
  • @ikkachu Thank you. I'm not trying to be unnecessarily nitpicky, I'm really not understanding this. You said that "If there's only one process, there's no question of ordering". But in `ls file.txt another_file.txt >out_err.txt 2>out_err.txt`, there's only one process and a question of ordering (presumably, because the content of `out_err.txt` is what it is). – Sweet Shell O'Mine Feb 07 '23 at 15:51
  • @SweetShellO'Mine, no, the order of writes is clear. It's just with two separate redirections, it writes them to the same place so that the second overwrites the first. But if it writes the error message first, that one goes first and gets overwritten, not the other way around. If you have multiple processes running at the same time, they might get to do the writes in whatever order, essentially randomly. – ilkkachu Feb 07 '23 at 16:05
  • Thank you very much! – Sweet Shell O'Mine Feb 07 '23 at 16:44
6

Simple redirection open(2)s the file with option O_CREAT and O_TRUNC this creates an empty file and positions the fileposition at the first byte

Appending a file opens it with with the O_APPEND option, this causes a seek to the current end of file before each write operation.

from man 2 open

    O_APPEND
          The file is opened in append mode.  Before  each  write(2),  the
          file  offset  is  positioned  at the end of the file, as if with
          lseek(2).  The modification of the file offset and the write op‐
          eration are performed as a single atomic step.

In other words append is guaranteed by the kernel to not clash

Jasen
  • 3,715
  • 13
  • 14