Simple rate limit is not enough because nmap increases scan delay when it hits rate limit. Here is what you can do best with iptables.
First create ipset lists
ipset create port_scanners hash:ip family inet hashsize 32768 maxelem 65536 timeout 600
ipset create scanned_ports hash:ip,port family inet hashsize 32768 maxelem 65536 timeout 60
And iptables rules
iptables -A INPUT -m state --state INVALID -j DROP
iptables -A INPUT -m state --state NEW -m set ! --match-set scanned_ports src,dst -m hashlimit --hashlimit-above 1/hour --hashlimit-burst 5 --hashlimit-mode srcip --hashlimit-name portscan --hashlimit-htable-expire 10000 -j SET --add-set port_scanners src --exist
iptables -A INPUT -m state --state NEW -m set --match-set port_scanners src -j DROP
iptables -A INPUT -m state --state NEW -j SET --add-set scanned_ports src,dst
How this works:
Here we store scanned ports in scanned_ports set and we only count newly scanned ports on our hashlimit rule. If a scanner send packets to 5 different port(see --hashlimit-burst 5) that means it is a probably scanner so we will add it to port_scanners set.
Timeout of port_scanners is the block time of scanners(10 minutes in that example). It will start counting from beginning (see --exist) till attacker stop scan for 10 seconds (see --hashlimit-htable-expire 10000)
You can set these parameters to most proper values for you.
Be aware of that someone can make any IP blocked by just make scan as spoofing. I suggest you don't set block timeout too long.
Addition:
If you want to add a whitelist, create a whitelisted list
ipset create whitelisted hash:net
and change drop rule with that
iptables -A INPUT -m state --state NEW -m set --match-set port_scanners src -m set ! --match-set whitelisted src -j DROP