sed in-place line deletion on full filesystem?

Question

Due to an application bug as yet undiagnosed, I have several hundred servers with a full disk. There is one file that has been filled up with duplicate lines—not a log file, but a user environment file with variable definitions (so I can't just delete the file).

I wrote a simple sed command to check for the erroneously added lines and delete them, and tested it on a local copy of the file. It worked as intended.

However, when I tried it on the server with the full disk, I got approximately the following error (it's from memory, not copy and paste):

sed: couldn't flush /path/to/file/sed8923ABC: No space left on deviceServerHostname

Of course, I know there's no space left. That's why I'm trying to delete stuff! (The sed command I'm using will reduce a 4000+ line file to about 90 lines.)

My sed command is just sed -i '/myregex/d' /path/to/file/filename

Is there a way I can apply this command despite the full disk?

(It must be automated, since I need to apply it to several hundred servers as a quick-fix.)

_{(Obviously the application bug needs to be diagnosed, but in the meantime the servers aren't working correctly....)}

---

Update: The situation I faced was resolved by deleting something else that I found out I could delete, but I'd still like the answer to this question, which would be helpful in the future and for other people.

/tmp is a no-go; it's on the same filesystem.

Before I freed up disk space, I did test and find out that I could delete the lines in vi by opening the file and running :g/myregex/d and then successfully save the changes with :wq. It seems it should be possible to automate this, without resorting to a separate filesystem to hold a temp file.... (?)

Answer 1

The -i option doesn't really overwrite the original file. It creates a new file with the output, then renames it to the original filename. Since you don't have room on the filesystem for this new file, it fails.

You'll need to do that yourself in your script, but create the new file on a different filesystem.

Also, if you're just deleting lines that match a regexp, you can use grep instead of sed.

grep -v 'myregex' /path/to/filename > /tmp/filename && mv /tmp/filename /path/to/filename

In general, it's rarely possible for programs to use the same file as input and output -- as soon as it starts writing to the file, the part of the program that's reading from the file will no longer see the original contents. So it either has to copy the original file somewhere first, or write to a new file and rename it when it's done.

If you don't want to use a temporary file, you could try caching the file contents in memory:

file=$(< /path/to/filename)
echo "$file" | grep -v 'myregex' > /path/to/filename

Answer 2

That's how sed works. If used with -i (in place edit) sed creates a temporary file with the new contents of the processed file. When finished sed, replaces the current working file with the temporary one. The utility does not edit the file in-place. That's exact the behavior of every editor.

It's like you perform the following task in a shell:

sed 'whatever' file >tmp_file
mv tmp_file file

At this point sed, tries to flush the buffered data to the file mentioned in the error message with the fflush() system call:

For output streams, fflush() forces a write of all user-space buffered data for the given output or update stream via the stream's underlying write function.

---

For your problem, I see a solution in mounting a separte filesystem (for instance a tmpfs, if you have enough memory, or an external storage device) and move some files there, process them there, and move them back.

Answer 3

Since posting this question I've learned that ex is a POSIX-compliant program. It's almost universally symlinked to vim, but either way, the following is (I think) a key point about ex in relation to filesystems (taken from the POSIX specification):

This section uses the term edit buffer to describe the current working text. No specific implementation is implied by this term. All editing changes are performed on the edit buffer, and no changes to it shall affect any file until an editor command writes the file.

"...shall affect any file..." I believe that putting something on the filesystem (at all, even a temp file) would count as "affecting any file." Maybe?*

Careful study of the POSIX specifications for ex indicate some "gotchas" about its intended portable use when compared to common scripted uses of ex found online (which are littered with vim-specific commands.)

1. Implementing +cmd is optional according to POSIX.
2. Allowing multiple -c options is also optional.
3. The global command :g "eats" everything up to the next non-escaped newline (and therefore runs it after each match found for the regex rather than once at the end). So -c 'g/regex/d | x' only deletes one instance and then exits the file.

So according to what I've researched, the POSIX-compliant method for in-place editing a file on a full filesystem to delete all lines matching a specific regex, is:

ex -sc 'g/myregex/d
x' /path/to/file/filename

This should work providing you have sufficient memory to load the file into a buffer.

_{*If you find anything which indicates otherwise, please, mention it in the comments.}

Answer 4

Use the pipe, Luke!

Read file | filter | write back

sed 's/PATTERN//' BIGFILE | dd of=BIGFILE conv=notrunc

in this case sed doesn't create a new file and just send output piped to dd which opens the same file. Of course one can use grep in particular case

grep -v 'PATTERN' BIGFILE | dd of=BIGFILE conv=notrunc

then truncate the remaining.

dd if=/dev/null of=BIGFILE seek=1 bs=BYTES_OF_SED_OUTPUT

Answer 5

You can truncate the file quite easily if you can get the byte count to your offset and your lines occur from a start point through to the end.

o=$(sed -ne'/regex/q;p' <file|wc -c)
dd if=/dev/null of=file bs="$o" seek=1

---

Or else if your ${TMPDIR:-/tmp} is on some other file system perhaps:

{   cut -c2- | sed "$script" >file
} <file <<FILE
$(paste /dev/null -)
FILE

Because (most) shells put their here-documents there in a deleted temp-file. It is perfectly safe so long as the <<FILE descriptor is maintained from start to finish and ${TMPDIR:-/tmp} has as much space as you need.

Shells which don't use temp files use pipes, and so are not safe to use this way. These shells are typically ash derivatives like busybox, dash, BSD sh - zsh, bash, ksh, and the Bourne shell, however, all use temp files.

apparently I wrote a little shell program last July to do something very like this

---

If /tmp is not viable, then so long as you can fit the file in memory something like...

sed 'H;$!d;x' <file | { read v &&
sed "$script" >file;}

...as a general case would at least ensure that the file was fully buffered by the first sed process before attempting to truncate the in/out file.

A more targeted - and efficient - solution could be:

sed '/regex/!H;$!d;x' <file|{ read v && cat >file;}

...because it wouldn't bother buffering lines you meant to delete anyway.

A test of the general case:

{   nums=/tmp/nums
    seq 1000000 >$nums
    ls -lh "$nums"
    wc -l  "$nums"
    sed 'H;$!d;x' <$nums | { read script &&  ### read always gets a blank
    sed "$script" >$nums;}
    wc -l  "$nums"
    ls -lh "$nums"
}

---

-rw-r--r-- 1 mikeserv mikeserv 6.6M Dec 22 20:26 /tmp/nums
1000000 /tmp/nums
1000000 /tmp/nums
-rw-r--r-- 1 mikeserv mikeserv 6.6M Dec 22 20:26 /tmp/nums

Answer 6

As noted in other answers, sed -i works by copying the file to a new file in the same directory, making changes in the process, and then moving the new file over the original.  That's why it doesn't work.  ed (the original line editor) works in a somewhat similar manner, but, last time I checked, it uses /tmp for the scratch file.  If your /tmp is on a different filesystem from the one that's full, ed may do the job for you.

Try this (at your interactive shell prompt):

$ ed /path/to/file/filename
P
g/myregex/d
w
q

The P (which is a capital P) is not strictly necessary.  It turns on prompting; without it, you're working in the dark, and some people find this disconcerting.  The w and q are write and quit.

ed is notorious for cryptic diagnostics.  If at any point it displays anything other that the prompt (which is *) or something that is clearly a confirmation of successful operation (especially if it contains a ?), do not write the file (with w).  Just quit (q).  If it doesn't let you out, try saying q again.

If your /tmp directory is on the filesystem that is full (or if its filesystem is full, also), try to find some space somewhere.  chaos mentioned mounting a tmpfs or an external storage device (e.g., a flash drive); but, if you have multiple filesystems, and they are not all full, you can simply use one of the other existing ones.  chaos suggests copying the file(s) to the other filesystem, editing them there (with sed), and then copying them back.  At this point, that may be the simplest solution.  But an alternative would be to create a writable directory on a filesystem that has some free space, set environment variable TMPDIR to point to that directory, and then run ed.  (Disclosure: I'm not sure whether this will work, but it can't hurt.)

Once you get ed working, you can automate this by doing

ed filename << EOF
g/myregex/d
w
q
EOF

in a script.  Or printf '%s\n' 'g/myregex/d' w q | ed -s filename, as suggested by don_crissti.

Answer 7

This answer borrows ideas from this other answer and this other answer but builds on them, creating an answer that is more generally applicable:

num_bytes=$(sed '/myregex/d' /path/to/file/filename | wc -c)
sed '/myregex/d' /path/to/file/filename 1<> /path/to/file/filename
dd if=/dev/null of=/path/to/file/filename bs="$num_bytes" seek=1

The first line runs the sed command with output written to standard output (and not to a file); specifically, to a pipe to wc to count the characters.  The second line also runs the sed command with output written to standard output, which, in this case is redirected to the input file in read/write overwrite (no truncate) mode, which is discussed here.  This is a somewhat dangerous thing to do; it is safe only when the filter command never increases the amount of data (text); i.e., for every n bytes that it reads, it writes n or fewer bytes.  This is, of course, true for the sed '/myregex/d' command; for every line that it reads, it writes the exact same line, or nothing.  (Other examples: s/foo/fu/ or s/foo/bar/ would be safe, but s/fu/foo/ and s/foo/foobar/ would not.)

For example:

$ cat filename
It was
a dark and stormy night.
$ sed '/was/d' filename 1<> filename
$ cat filename
a dark and stormy night.
night.

because these 32 bytes of data:

I  t     w  a  s \n  a     d  a  r  k     a  n  d     s  t  o  r  m  y     n  i  g  h  t  . \n

got overwritten with these 25 characters:

a     d  a  r  k     a  n  d     s  t  o  r  m  y     n  i  g  h  t  . \n

leaving the seven bytes night.\n left over at the end.

Finally, the dd command seeks to the end of the new, scrubbed data (byte 25 in this example) and removes the rest of the file; i.e., it truncates the file at that point.

---

If, for any reason, the 1<> trick doesn’t work, you can do

sed '/myregex/d' /path/to/file/filename | dd of=/path/to/file/filename conv=notrunc

---

Also, note that, as long as all you’re doing is removing lines, all you need is grep -v myregex (as pointed out by Barmar).

Answer 8

I can imagine that in that moment, you have something/enough free RAM. You could use tmpfs and work with file there, using sed.

I think that /path/to/file/filename is not big (they are just user environment file ).

mkdir /mnt/ramdisk
mount -t tmpfs -o size=1m tmpfs /mnt/ramdisk

Test it with df -h /mnt/ramdisk or mount, or exit status code

echo $?
0

df -h /tmp/tmpfs/
Filesystem      Size  Used Avail Use% Mounted on
tmpfs           1.0M     0  1.0M   0% /tmp/tmpfs

Now you can move or copy the file to /mnt/ramdisk and do your stuff, then replace original

More about tmpfs or ramfs on jamescoyle site.