I'm not understanding setgid on executables on my platform (ubuntu). g-x,g+s is not giving the process effective group permissions of the program's owner.
$ gcc perms.c -o perms; ls -l ; ./perms
-rwxr-xr-x 1 ubuntu ubuntu 9302 Feb 24 01:00 perms*
-rw-r--r-- 1 ubuntu ubuntu 1437 Feb 21 08:41 perms.c
ruid: ubuntu:1000 group:1000
euid: ubuntu:1000 group:1000
$ sudo useradd alice; groups alice $USER
alice : alice
ubuntu : ubuntu sudo rvm
$ chmod g+s ./perms; ls -l ./perms ; sudo -u alice ./perms
rwxr-sr-x 1 ubuntu ubuntu 9302 Feb 24 01:00 perms*
ruid: alice:1001 group:1002
euid: alice:1001 group:**1000**
This is all expected, the problem is:
$ chmod g-x ./perms; ls -l ./perms ; sudo -u alice ./perms
-rwxr-Sr-x 1 ubuntu ubuntu 9302 Feb 24 01:00 perms*
ruid: alice:1001 group:1002
euid: alice:1001 group:**1002**
My understanding is that g+s gives the process the effective group id of the program's owner, but it's not. Obviously, this is because of the g-x since that was the only change.
Edit1: removed the section on how g-x,g+s works on directories, because people were thinking I was asking about S on directories. I'm not.
Edit2: since I'm getting condesention from one answer. This is also different than from what happens with u-x,u+s:
$ rm -rf perms temp/; gcc perms.c -o perms
$ chmod ug-x,ug+s ./perms; ls -l ./perms; sudo -u alice ./perms
-rwSr-Sr-x 1 ubuntu ubuntu 9302 Feb 24 21:22 ./perms*
ruid: alice:1001 group:1002
euid: ubuntu:1000 group:1002
Here u-x,u+s is respected, but g-x,g+s is not.
My question: why is capital S sgid ignored for executables?
g-x,g+s is respected on directories.
u-x,u+s is respected on executables.
But g-x,g+s is not respected on executables.
WHY?
ANSWER: my research seems to have dead-ended at "because System V arbitrarily decided it means something else."