I am more interested in sockets than regular files, but basically I want to know whether one process can "see" a socket as blocking where another process can see it as non-blocking. I am guessing yes, and that the kernel handles all of this depending on what options were used in the syscall.
I guess this would be more about Unix domain sockets than TCP sockets since I don't think 2 different processes could use the same TCP socket (but I could be wrong)
You guess wrong.
The only property which is per-file descriptor and which can be changed with fcntl(F_SETFD) is the FD_CLOEXEC close-on-exec flag.
All the other properties are either per file object ("open file description" in POSIX lingo -- which can be changed with fcntl(F_SETFL)), or per inode.
Setting the non-blocking flag with fcntl(F_SETFL, | O_NONBLOCK) or with ioctl(FIONBIO) will affect all the file descriptors that refer to that open file. There is also no way to make a file non-blocking only for reading or writing.
This is far from ideal -- you can also refer to this Q&A on StackOverflow, especially the link to the lkml discussion about a failed attempt to fix it somehow.
Notice that regular files are basically non-blocking -- a poll(2) or select(2) on them will return immediately.
If you're only interested in sockets, you should use send(2) or recv(2) with the MSG_DONTWAIT flag instead of read(2) or write(2). Contrary to what you say, a socket file descriptor can be shared between processes, and no matter what its family/protocol/options are. And that also applies to a listening socket.
Changing one file descriptor in a process to blocking or non-blocking does not affect other file descriptors of the same or other processes. O_NONBLOCK is a property of a file descriptor, not of the file or socket. I misremembered. To avoid that one has to open files twice (instead of dup/fork) or create sockets twice.
And different processes can use the same TCP socket. See this question. They can even write into the same socket, if they are synchronized correctly.
