See the source code, specifically libmisc/chkname.c. Shadow is pretty conservative: names must match the regexp [_a-z][-0-9_a-z]*\$? and may be at most GROUP_NAME_MAX_LENGTH characters long (configure option, default 16; user names can usually go up to 32 characters, subject to compile-time determination).
Debian relaxes the check a lot. As of squeeze, anything but whitespace and : is allowed. See bug #264879 and bug #377844.
POSIX requires allowing letters of either case, digits and ._- (like in file names). POSIX doesn't set any restriction if you don't care about portability. A number of recommended restrictions come from usage:
- Colons, newlines and nulls are right out; you just can't use them in
/etc/passwd or /etc/group.
- An name consisting solely of digits is a bad idea —
chown and chgrp are supposed to treat a digit sequence as a name if it's in the user/group database, but other applications may treat any number as a numerical id.
- An initial
- or a . in a user name is strongly not recommended, because many applications expect to be able to pass $user.$group to an external utility (e.g. chown $user.$group /path/to/file)¹. A . in a group name should cause less trouble, but I'd still recommend against it.
/ is likely to cause trouble too, because some programs expect to be able to use user names in file names.
- Any character that the shell would expand is probably risky.
- Non-ASCII characters should be ok if you don't care about sharing with systems that may use different encodings.
¹ All modern implementations expect chown $user:$group, but support chown $user.$group for backward compatibility, and there are too many applications out there that pass a dot to remove that compatibility support.