How do I trim bytes from the beginning and end of a file?

Question

I have a file, that has trash (binary header and footer) at the beginning and end of the file. I would like to know how to nuke these bytes. For an example, let's assume 25 bytes from the beginning. And, 2 bytes from the end.

I know I can use truncate and dd, but truncate doesn't work with a stream and it seems kind of cludgey to run two commands on the hard file. It would be nicer if truncate, knowing how big the file was, could cat the file to dd. Or, if there was a nicer way to do this?

score 24 · Accepted Answer · edited May 24 '17 at 21:26

24

You can combine GNU tail and head:

tail -c +26 file | head -c -2

will output the contents of file starting at byte 26, and stopping two bytes (minus two -2) before the end. (-c operates on bytes, not characters.)

edited May 24 '17 at 21:26

Evan Carroll

28,578
45
164
290

answered May 24 '17 at 20:55

Stephen Kitt

411,918
54
1,065
1,164

3

Note that `tail -c +26` is standard (`head -c -2` is not). – Stéphane Chazelas May 24 '17 at 21:47
1

@StéphaneChazelas is there a POSIX compliant version of the `head -c -2` command? – cat May 25 '17 at 01:30
This does not work well when the file contains null bytes, right? – Zorglub29 Mar 05 '20 at 21:47
@Zorglub29 it works for me even with null bytes. – Stephen Kitt Mar 05 '20 at 22:08

score 11 · Answer 2 · answered May 24 '17 at 20:58

11

dd will do both for you in a single command. Set the block size to 1 byte, skip the 25 first bytes, count to the size of file minus skip and end bytes.

100 byte file
file.img

dd if=./file.img of=./trimed_file.img bs=1 skip=25 count=73

Double check numbers cause it might count from 0.

answered May 24 '17 at 20:58

jc__

2,485
1
16
22

isn't too slow? – Ravexina May 24 '17 at 21:00
1

Too slow? Only if the file is very large like many GB. Then you could adjust the block size. – jc__ May 24 '17 at 21:01
it took 0m8.319s for a 1.3 MB file, For the count he can use something like `$(( `stat -c %s test` - 5 ))` I guess. – Ravexina May 24 '17 at 21:06
Remember that skip and count are blocks not bytes. Pick the largest multiple for the block size. The closer the block size is to the HD buffer size the more efficient and faster the command is. – jc__ May 24 '17 at 21:19
Yeah, however bs is defined 1 in your command :-) so in this case it will work... – Ravexina May 24 '17 at 21:22
5

with `bs=1` dd makes two system calls for each byte, that's horribly slow. GNU dd has the `count_bytes` and `skip_bytes` that can help, do something like `dd bs=8192 iflag=count_bytes,skip_bytes skip=25 count=73 < infile > outfile` – ilkkachu May 24 '17 at 21:25

score 2 · Answer 3 · answered May 24 '17 at 21:57

With ksh93:

{ head -c "$n"; } < file <#((n = EOF - 25 - 2 , 25))

Or to do it in-place:

{ head -c "$n"; } < file <#((n = EOF - 25 - 2 , 25)) 1<>; file

If you have /opt/ast/bin ahead of your $PATH, you'll get the head builtin.

<#((...)) is a lseek() operator. ... is interpreted as an arithmetic expression where EOF is the length of the file. So above, we're assigning the length of the portion to display to $n and seeking to 25 bytes within the file.
<>; is a open-in-read+write-mode-and-truncate-if-the-command-is-successful redirection operator.

How do I trim bytes from the beginning and end of a file?

3 Answers3