54
root@system:~# less myfile
-bash: /bin/less: Input/output error

The root filesystem is dead. But my cat is still alive (in my memory):

root@system:~# cat > /tmp/somefile
C^d
root@system:~#

He's kind of lonely though, all his friends are gone:

root@system:~# mount
-bash: /bin/mount: Input/output error
root@system:~# dmesg
-bash: /bin/dmesg: Input/output error
root@system:~# less
-bash: /bin/less: Input/output error
root@system:~# chmod
-bash: /bin/chmod: Input/output error

The system is still running, and fulfilling its purpose. I know, I know, the only sane response to this is to get the system down and replace the root drive. Unfortunately that's not a option as it would cost a lot of time and money. Also, it would kill my cat, and that would make me sad.

I've thought of bringing him his usual friends from a donor. I dare not try to scp them in, in case ssh tries to load it and cuts the line (the binary is gone anyway). This sounds like a job for my cat's cousin:

root@system:~# netcat -l 1234 > /tmp/less
-bash: netcat: command not found

Unfortunately he's long gone.

Now, I can try to trick my cat to perform a ritual to resurrect him:

cat > netcat < /dev/tcp/localhost/9999

And that sort of worked. He's almost alive:

root@system:/tmp# /tmp/netcat
-bash: /tmp/netcat: Permission denied

He just needs a tiny spark of life. That little +x magic incantation that I cannot recite at the moment.

Can you assist me bringing my cat's friends back?

Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
loopbackbee
  • 4,442
  • 3
  • 24
  • 30
  • 3
    Can you run `/lib/ld-linux.so.2 ./netcat` (or your system's equivalent) and get something going? – Michael Homer Oct 14 '18 at 03:48
  • 4
    Also: what operating system is this? Do you have any vfat or NTFS filesystems mounted? Network filesystems? Partitions you can erase the contents of? Are there *any* accessible files with execute permission? – Michael Homer Oct 14 '18 at 03:56
  • 3
    "Also, it would kill my cat, and that would make me sad." – I always thought the saying "No kiddens were harmed during the production of this" was just a saying, but well… – rugk Oct 14 '18 at 16:13
  • 7
    I totally had to upvote this for all these cat analogies… – rugk Oct 14 '18 at 16:13
  • It's probably better to exfiltrate whatever remains accessible in that machine (your cat? ☺) using `cat > /dev/tcp/othermachine` – Ángel Oct 14 '18 at 16:41
  • @MichaelHomer It's Debian Linux. There's working blockdevices attached, but nothing mounted. tmpfs is explicitly mounted without noexec, so executing things from there is no problem – loopbackbee Oct 14 '18 at 16:43
  • 2
    poor cat, i hope he's okay :( – cat Oct 14 '18 at 20:53
  • 2
    It's probably only a matter of time before the broken file system cause some critical process to fail and you will have to reboot. During the next reboot (whatever the reason may be) there is a significant risk the system doesn't come up again. Better plan for a maintenance window before you have to scramble to get the system back online. – kasperd Oct 14 '18 at 21:17
  • Your cat's parents will probably panic and stop working soon. – Nonny Moose Oct 14 '18 at 22:32
  • Same thing happened to me the other day after I restarted following a power failure. The machine was my internet firewall and it continued to do its job just fine. When I ran `ls -l` in several directories, many entries were corrupt. I thought the machine was toast and would need rebuilding. I tried rebooting with a forced fsck and everything came right. Well maybe: I fear there is still something corrupt lurking in the file-system somewhere waiting to strike. I'm going to have to replace that cheap Chinese computer... `Linux 4.14.65-gentoo #1 SMP x86_64` – Adrian Pronk Oct 17 '18 at 18:24

1 Answers1

36

There are several possibilities, all depending on the exact parameters of your situation right now. I'm going to assume Linux in the following examples where applicable, but similar functionality exists on other platforms in most cases.

  • You might be able to get the dynamic loader to run an executable for you. Assuming cat is dynamically-linked, your platform's equivalent of /lib/ld-linux.so.2 will likely also be in memory and thus usable to run a binary:

    $ /lib64/ld-linux-x86-64.so.2 ./chmod
    chmod: missing operand
    

    You may have multiple of these (32- and 64-bit are likely) and there may be multiple copies available, or symlinks that need resolving. One of those may work.

  • If you have a mounted vfat or NTFS filesystem, or another that treats all files as 777, you can create your executable on there.

    $ cat > /mnt/windows/chmod < /dev/tcp/localhost/9999
    
  • If you have a mounted network filesystem, even if it's not locally writable, you can create files on the remote system and use those normally.
  • If there's a mounted partition you don't care about the contents of, on a drive that is still mostly working, you can replace the contents with a new image of the same filesystem type containing executables you want - cat should be fine for this in the role people usually use dd for, and you can provide the image over the network.

    $ cat > /dev/sdb1 < ...
    

    This one is plausible, but has a lot of places not to work depending on what exactly is still in memory from that partition.

  • If there is any accessible file that has execute permission on any writable filesystem, you can cat > into it to replace the contents with a binary of your choosing.

    $ cat > ~/test.py < ...
    
  • Since Bash is still running, you could dynamically load a Bash plugin into the process that exposes chmod. In particular, you could install and load ctypes.sh, which provides a foreign function interface to Bash, and then dlcall chmod ./netcat 511.
  • You could bring in a dynamic library file foo.so of your construction and then have cat load it on your behalf by way of LD_PRELOAD, allowing you to execute arbitrary code.

    $ LD_PRELOAD=./hack.so cat /dev/null
    

    If you intercept, for example, open:

    int open(const char *path, int flags, ...) {
        chmod(path, 0755);
        return -1;
    }
    

    then you can do whatever you need to do in there.

My suggestion would be to bring in a statically-linked busybox executable as the first item (or really, only item) so that you've got the full range of commands available without reusing whatever hack got you to that point to exhaustion.

Michael Homer
  • 74,824
  • 17
  • 212
  • 233
  • 3
    Re "statically-linked `busybox`": Note that [`sash`](https://en.wikipedia.org/wiki/Stand-alone_shell) was specifically designed for this kind of use-case and may be more readily available in a pinch (e.g. you could install it in advance, and leave a copy running somewhere so it's in memory when the system breaks... or you can ditch your pets and buy a bunch of cattle instead, I suppose). – Kevin Oct 14 '18 at 20:11