Depending on your users, that may be a good or bad idea.
The su command can be used by ordinary users to log into any other account, not just the superuser's, provided they know the password of the account.
This is useful e.g. when two users are cooperating on a project, one of them (userA) is logged in and they need to read a file only accessible to the other user (userB). Simply running
su -c 'cat /home/userB/file' userB >file
can be used to copy the file to the logged in user's home directory, but that is only possible if userA is allowed to run the su command.
On PostgreSQL database servers, it is usually possible for database administrators to switch to the postgres user to perform some maintenance tasks that are not possible while the database server is live; for this to work, they need to be able to run su - postgres.
The same considerations apply to the sudo command (which is different and way more complex than su -- it is only the default configuration that makes the command primarily useful for the sudoers group, because that group is allowed to run any command as root. However, if you add any custom rules, e.g. allowing any user to run updatedb with no arguments, then users outside the sudoers group need execute permission on sudo.
Also, there is not only su that is setuid -- there is also
sg (allows temporarily joining a group if there is a group password set)
passwd (allows changing the password)
chfn (allows changing user information)
chsh (allows changing default shell)
and several others. These tools share lots of code with the su command, so changing the mode of /bin/su alone will not buy you much, and changing the mode of all of them will certainly annoy your users, especially in the case of passwd.