In iptables-extensions(8) the set module is described and it is discussed that it is possible to react to the presence or absence of an IP or more generally a match against an IP set.
However, it does not seem that there is a way to append items to an IP set on the fly using an iptables rule.
The idea being that if I use the recent module, I could then temporarily blacklist certain IPs that keep trying and add them into an IP set (which is likely faster). This would mean less rules to traverse for such cases and matching against an IP set is said to be faster as well.