1

I have a Linux box, and am able to login using ssh. My shell is /bin/bash, I end up at a bash shell prompt.

I can execute bash built-in commands like echo. I am able to execute some commands from (prefixing path) /bin like ls /bin

Other commands like whoami appear to hang. I CTRL-C and return to the bash prompt.

And other commands get completely hung, CTRL-C does not exit the command and I do not return to the bash prompt. ls /usr/bin is an example of a command that has hung the shell.

here is the path... because I know someone will ask about it...

$ echo $PATH  
/u01/jdk/bin:/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/u01/app/oracle/tools/home/oracle/bin

What would cause such a situation? What can I do to resolve the problem?

SouravGhosh
  • 553
  • 5
  • 12
vince kraemer
  • 11
  • 1
  • 3
  • 1
    What is your default login shell? `getent passwd "$LOGNAME"` or `cat /etc/passwd | grep "$USER"` – jesse_b Dec 21 '17 at 16:33
  • updated q with answer to Jesse_b question. – vince kraemer Dec 21 '17 at 16:45
  • 1
    Do you administer this machine? If so, does `dmesg` contain anything about errors — particularly disk errors (start at the bottom and read upwards — the top may have a bunch of unimportant boot messages). (If someone else administers it, I doubt there is much you can do to fix it. At least if it's disk errors). – derobert Dec 21 '17 at 16:49
  • 3
    we need more information, it could also be something like a hung or slow LDAP server, or ... (try stracing things, or equivalent?) – thrig Dec 21 '17 at 16:52
  • 2
    If it's everything that needs to read usernames (username-uid mappings), hung LDAP server or such; if it's everything related to a certain path, a hung network drive is one possibility; if it's totally everything, then it's possible you have an unbearable number of tasks in the run queue, and/or a huge I/O load. Or something completely different. – ilkkachu Dec 21 '17 at 17:27
  • For a related question, see https://unix.stackexchange.com/questions/412192/ . – JdeBP Dec 21 '17 at 17:29
  • 1
    once upon a time booting into single user mode (the rescue shell was also single user) would behave like that. Are you sure you are not in single user mode? – Tomáš Pospíšek Dec 21 '17 at 19:49
  • It could be a problem with user space. Or maybe it's a physical problem, if `ls` is able to work with `/bin` and not `/usr/bin`. – Tanner Babcock Dec 21 '17 at 22:49
  • Is /u01 an nfs mount, by any chance? – Jeff Schaller Dec 22 '17 at 00:26
  • "commands like whoami appear to hang". Knowing the other commands affected would help identify a common factor. – steve Dec 22 '17 at 00:53

1 Answers1

1

derobert already mentioned dmesg in the comments. Try running it using an absolute path, like this:

/bin/dmesg | /bin/more

See if there are anything that would seem like a disk error: it might be that one of the directories listed in your PATH is on a disk that's failing. If the disk is failing in a way that causes it to get stuck, forever retrying the failing read operation, that might be causing the shell to hang when it is searching for a command to execute.

You could also try just minimizing the PATH: initially just set

PATH=/bin:/sbin

If you now can run commands without hanging and without using absolute paths, add back /usr/bin and /usr/sbin too. 

PATH=/bin:/usr/bin:/usr/sbin:/sbin

Now you should have all the standard commands available to you without using absolute paths. If the hanging phenomenon does not re-occur, you can now start checking the remaining directories.

As a quick check, you can use the ls command, but run it in the background so you won't cause your current session to become unusable if the command hangs. For example:

ls /u01/jdk/bin &

If you see the ls command output, just press Enter once to see the prompt again. (You will probably also see a message from the shell indicating that a backgrounded command did just complete.)

If the output of ls won't appear or is clearly incomplete, check the output of the dmesg command again: you might find the kernel is now spewing disk I/O errors to the end of the output. Those error messages could be used to identify the failing disk device, but by knowing exactly at which path the ls command failed, you may already have a fairly good idea of that.

telcoM
  • 87,318
  • 3
  • 112
  • 232