What you are asking is a bit broad, so I will try to stick to the general terms. To rephrase it, you would like to have an idea over time of your network usage broken down by process AND by protocol.
For domestic use, try ntop. It will do more than you probably need, and to top if off, it shows historical graphics.
http://www.ntop.org/products/traffic-analysis/ntop/
There are tools that fit that job temporarily. For a general glance of what is happening, nethogs breaks down the applicational usage of network quite nicely.
http://nethogs.sourceforge.net
"NetHogs is a small 'net top' tool. Instead of breaking the traffic down per protocol or per subnet, like most tools do, it groups bandwidth by process. "
You also have dtrace4linux and sysdig, which can do very surprising things for accounting almost all aspects of server usage. There is a very interesting book about dtrace "Systems Performance: Enterprise and the Cloud" from Brendan Gregg.
For global accounting of traffic usage, you can either sample ifconfig, or the rx/tx transferred bytes in proc. If managing multiple servers, better use an NMS and SNMP.
If managing a network of servers, you would want to do iptables accounting in the router, or setup Netflow in your firewall or router, and listen and store the data in a DB. nfSen is particularly useful to visualize that kind of that.
As for security related attacks, IDS is a very interesting technology. You can run Snort in your box, however it is generally ran in a box that can listen to the traffic flowing.