I know of strace, which is super handy for watching system calls. Is there an equivalent that can trace calls to external libraries?
For example, I'm trying to debug some SSLeay misbehavior with a third-party binary that I cannot access the code for (and who's developer is unresponsive). I believe I have nailed down the problem, but I can't test this without seeing exactly how the data being signature-checked is being formatted for sending off to SSLeay.