39

I've got an extreme problem, and all of the solutions I can imagine are complicated. According to my UNIX/Linux experience there must be an easy way.

I want to delete the first 31 bytes of each file in /foo/. Each file is long enough. Well, I'm sure somebody will deliver me a suprisingly easy solution I just can't imagine. Maybe awk?

Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
von der tann
  • 393
  • 1
  • 3
  • 5
  • 2
    Any awk/sed/ed solution will be line-oriented, so if you don't know the first line will be at least 31 characters then complications ensue. – glenn jackman May 27 '11 at 15:51

5 Answers5

33
for file in /foo/*
do
  if [ -f "$file" ]
  then
    dd if="$file" of="$file.truncated" bs=31 skip=1 && mv "$file.truncated" "$file"
  fi
done

or the faster, thanks to Gilles' suggestion:

for file in /foo/*
    do
      if [ -f $file ]
      then
        tail +32c $file > $file.truncated && mv $file.truncated $file
      fi
    done

Note: Posix tail specify "-c +32" instead of "+32c" but Solaris default tail doesn't like it:

   $ /usr/bin/tail -c +32 /tmp/foo > /tmp/foo1
    tail: cannot open input

/usr/xpg4/bin/tail is fine with both syntaxes.

If you want to keep the original file permissions, replace

... && mv "$file.truncated" "$file"

by

... && cat "$file.truncated" "$file" && rm "$file.truncated"
jlliagre
  • 60,319
  • 10
  • 115
  • 157
  • 1
    Suggesting `dd` here is overkill, `tail` is more appropriate (simpler, less risk of a killer typo, no spurious messages on stderr). – Gilles 'SO- stop being evil' May 27 '11 at 22:16
  • You are right. I usually avoid commands intended to process text files when processing possibly binary ones but "tail +32c" will work here. – jlliagre May 28 '11 at 08:26
  • 1
    @jlliagre: You have written `cut` (shouldn't that be tail? ... asis, it doesn't work for me... – Peter.O May 28 '11 at 09:21
  • Of course, it's tail. Sorry about the mismatch. – jlliagre May 28 '11 at 09:23
  • @jlliagre: On Solaris, you should have `/usr/xpg4/bin` ahead of `/usr/bin` on your `PATH`, or you'll be stuck in the early 1990s. Many unices (e.g. GNU, BusyBox) no longer support the historical `+32c` syntax, and take it to mean a file called `+32c` (as POSIX requires). – Gilles 'SO- stop being evil' May 28 '11 at 12:31
  • @Gilles: I should indeed as I usually strongly support standard conformance but in that specific case, I'm afraid I won't switch soon. What refrains me to do it is that change would trigger a gazillion of issues with existing scripts, and would distance me from the community of Solaris users who in their very large majority are still using the traditional commands. – jlliagre May 29 '11 at 06:51
  • tail: cannot open '+32c' for reading: No such file or directory – Aryeh Armon Aug 02 '17 at 14:02
  • @AryehArmon As stated in my reply, you can use instead "tail -c +32 $file" on POSIX systems. – jlliagre Aug 02 '17 at 20:26
  • @jlliagre this will give me last 32 bytes, correct? – Aryeh Armon Aug 03 '17 at 07:15
  • @AryehArmon Incorrect, unless if the file size is exactly 64 bytes. – jlliagre Aug 03 '17 at 08:25
  • 1
    `mv` changes the file permissions. you can use `cat` instead of `mv` like this: ```tail +32c $file > $file.truncated && cat $file.truncated > $file && rm $file.truncated``` @jlliagre – Hameda169 Aug 30 '20 at 18:36
  • @Hameda169 Thanks for your comment,suggestion added. Note that technically, it is not `mv` that changes permissions but the first redirection that sets potentially different ones to the intermediary one. `mv` keeps the source file permissions. – jlliagre Aug 30 '20 at 20:42
18

The following commands cut first 31 bytes from $file (using $file~ as a temp. copy):

dd if="$file" of="$file~" bs=1 skip=31
mv "$file~" "$file"

You only need to list or find all files under /foo/ and execute the two above for every $file found.

alex
  • 7,093
  • 6
  • 28
  • 30
15

tail -c +32 outputs its input minus the first 31 bytes. (Yes, the argument is off by one.) To edit a file in place, use sponge in a loop, or if you don't have it and don't want to bother, do its job in the shell:

for x in /foo/*; do tail -c +32 "$x" | sponge "$x"; done
for x in /foo/*; do tail -c +32 "$x" >"$x.new" && mv "$x.new" "$x"; done

If the commands are interrupted for whatever reason (e.g. a power failure), it might be hard to figure out where you left off. Writing the new files to a separate directory would make things easier.

mkdir /foo.tmp
cd /foo
for x in *; do tail -c +42 -- "$x" >"/foo.tmp/$x" && rm -- "$x"; done
mv /foo.tmp/* /foo
rmdir /foo.tmp

If the files are really large (as in, large enough that having two copies of even a single one is a problem), you can use one of the techniques mentioned in this thread.

Community
  • 1
Gilles 'SO- stop being evil'
  • 807,993
  • 194
  • 1,674
  • 2,175
2

You can use Vim in Ex mode:

for each in /foo/*
do
  ex -sc '%!tail -c+32' -cx "$each"
done

  1. % select all lines

  2. ! run command

  3. x save and close

Zombo
  • 1
  • 5
  • 43
  • 62
1

Using dd with block size 1 is extremely slow. It is possible to use a large block size while specifying the skip in bytes like this:

dd iflag=skip_bytes if=infile.txt of=outfile.txt skip=31 bs=1M

jnalanko
  • 111
  • 2