There could be different reasons:
First of all I will assume that you ssh server is listening in the correct IP/interface, anyway an easy way to ensure it would be by typing in the ssh server:
netstat -putan |grep ssh
or depending on your OS type/version:
ss -putan |grep ssh
In case listening in the correct interface, check also if your SSH server is listening in tcp, tcp6 or both.
Sometimes, if you listen in both tcp and tcp6 (that would mean that you would be listening in IPv4 and IPv6), it can give you problems... You can only listen in IPv4 by editing the ssh configuration file /etc/ssh/sshd_config and adding/editting this value:
AddressFamily inet
After this change you would have to restart the SSH server.
Even if in this case you cannot access by SSH, I would recommend pinging the destination IP. If you receive no response, check IP/mask in both sides as well as the routing tables. The routing tables can be checked by the command:
ip route