I don't understand its real significance
Let's see how it's caused and used.
The major cause of SIGTRAP is the int3 instruction.
We can check that with:
int main() {
asm("int3");
return 0;
}
which when run outputs:
Trace/breakpoint trap (core dumped)
and has exit status 133 = 128 + 5, thus signal 5, SIGTRAP.
GDB inserts int3 instruction in the text segment, sets up ptrace, and lets the program run. When it hits int3, ptrace wakes up the parent which can the monitor the child's state.
Note that there are also hardware breakpoints, which have a different mechanism: What is the difference between hardware and software breakpoints? | Stack Overflow
More about int3
int3 has two encodings:
- the regular
int prefix + 3 which takes up 2 bytes, int 3 in NASM
- a special 1 byte long encoding,
int3 in NASM
The one byte long encoding is fundamental for GDB. If the instruction were larger than 1 byte, it could overwrite multiple instructions, which would be messy.