7

I have a problem with removing empty dir, strace shows error:

rmdir("empty_dir") = -1 ENOTEMPTY (Directory not empty)

And ls -la empty_dir shows nothing. So i connected to the fs (ext4) with debugfs and see the hidden file inside this dir:

# ls -lia empty_dir/
total 8
44574010 drwxr-xr-x 2 2686 2681 4096 Jan 13 17:59 .
44573990 drwxr-xr-x 3 2686 2681 4096 Jan 13 18:36 ..

debugfs:  ls empty_dir
 44574010  (12) .    44573990  (316) ..  
 26808797  (3768) _-----------------------------------------------------------.jpg

Why could this happen? And any chance to solve this problem without unmounting and full checking fs?

Additional information:

The "hidden" file is just a normal jpg file and can be opened by the image viewer:

debugfs:  dump empty_dir/_-----------------------------------------------------------.jpg /root/hidden_file

# file /root/hidden_file 
/root/hidden_file: JPEG image data, JFIF standard 1.02

rm -rf empty_dir is not working with the same error:

unlinkat(AT_FDCWD, "empty_dir", AT_REMOVEDIR) = -1 ENOTEMPTY (Directory not empty)

find empty_dir/ -inum 26808797 shows nothing.

bfuvx
  • 111
  • 7
  • 3
    Do you have any sort of kernel-level rootkit installed? That would do all kinds of mess like this. – Mukesh Sai Kumar Jan 13 '18 at 16:47
  • it can be due to special characters into a filename if so the good methods are renaming the file if it can be done from then wildcards ones ... or in case you are lucky & just want to remove the directory trying `rm -rf thatdirectory` most of time does the job. else the last method is to erase file from it's inode number . `find . -inum [inode-number] -exec rm -i {} \;` for example – francois P Jan 13 '18 at 17:19
  • debufs show a file still registered as confirmed by @Hauke Laging – francois P Jan 13 '18 at 17:41
  • 1
    @MukeshSaiKumar This is a backup server with a minimal numbers of processes and without public network access. There are not any unknown processes, kernel modules or malicious activities. I edited question with adding additional information. – bfuvx Jan 13 '18 at 17:54
  • @francoisP I added information about `find -inum` and `rm -rf`. – bfuvx Jan 13 '18 at 17:55
  • ah ok ... :) ... – francois P Jan 13 '18 at 17:59
  • Is it exported over NFS or any other network FS? – Stéphane Chazelas Jan 13 '18 at 19:17
  • @StéphaneChazelas Its exported over nfs but only with read-only access. Stopping/restarting nfs changes nothing, but! i can see this "hidden_file" from nfs client side just with `ls`. – bfuvx Jan 13 '18 at 19:40
  • Is the file _open_ by anything on any of the NFS clients? – Stéphane Chazelas Jan 13 '18 at 19:53
  • @StéphaneChazelas no, this file isn't open. I unmounted nfs client and situation didn't change. – bfuvx Jan 13 '18 at 21:19
  • 1
    @MukeshSaiKumar You was right about rootkit. I added answer. – bfuvx Jan 15 '18 at 00:16

3 Answers3

4

I straced ls and got more information to dig (stripped non-important syscalls):

open("empty_dir", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 3
getdents(3, /* 3 entries */, 32768)     = 80
write(1, ".\n", 2.)                     = 2
write(1, "..\n", 3..)                   = 3

Hmm, we see that syscall getdents works correctly and returned all 3 entries ('.','..' and '_---*'), but ls wrote only '.' and '..'. It means that we have some problem with wrapper around getdents which is used by coreutils. And coreutils use readdir glibc wrapper for getdents. Also to prove that there are no problems with getdents i tested little prog from example section of getdents' man page. This prog showed all files.

Maybe we just found a bug at the glibc? So i updated glibc package to the last version in my distro but didn't get any good result. Also i didn't find any correlating information in bugzilla.

So let's go deeper:

# gdb ls
(gdb) break readdir
(gdb) run
Breakpoint 1, 0x00007ffff7dfa820 in readdir () from /lib64/libncom.so.4.0.1
(gdb) info symbol readdir
readdir in section .text of /lib64/libncom.so.4.0.1

Wait, what? libncom.so.4.0.1? Not a libc? Yes, we just see a malicious shared library with libc functions for hiding malicious activity:

# LD_PRELOAD=/lib64/libc.so.6 find / > good_find
# find / > injected_find
# diff good_find injected_find
10310d10305
< /lib64/libncom.so.4.0.1
73306d73300
< /usr/bin/_-config
73508d73501
< /usr/bin/_-pud
73714d73706
< /usr/bin/_-minerd
86854d86845
< /etc/ld.so.preload

Removing rootkit files, checking all packages' files (rpm -Va in my case), auto-start scripts, preload/prelink configs, system files (find / + rpm -qf in my case), changing affected passwords, finding and killing rootkit's processes:

# for i in /proc/[1-9]*; do name=$(</proc/${i##*/}/comm); ps -p ${i##*/} > /dev/null || echo $name; done
_-minerd

In the end full system update, reboot and problem solved. Reason of the successful hacking: ipmi interface with very old firmware which suddenly was available from the public network.

bfuvx
  • 111
  • 7
  • Wow, you were nice enough to give a full disclosure and diagnosis here! +1 :) – Mukesh Sai Kumar Jan 15 '18 at 13:17
  • just curious, did "minerd" show up in 'ps' or any other tool? and how did you root cause it to ipmi? – rajaganesh87 Jan 18 '18 at 08:31
  • @rajaganesh87 _-minerd process was invisible for libproc based utils (ps, top, etc ...), but was visible via procfs (that's why i used for-loop with /proc/[1-9]* and `ps -p` for every pid for detection invisible processes). Hackers compiled malicious libproc-3.2.8.so and located it to the /lib/ dir (distro's libproc-3.2.8.so was in /lib64, but /lib dir in this distro has more priority when shared libs is loading). With hacked ipmi it's possible to reboot server. Reboot -> grub -> init=/bin/bash -> make backdoor -> reboot again to the normal system -> use backdoor for malicious activity. – bfuvx Jan 20 '18 at 00:16
1

Within debugfs you can delete the file. You do not even need the file name (which may be relevant if there are problems with special chars as francois P guessed in the comments):

kill_file <26808797>
Hauke Laging
  • 88,146
  • 18
  • 125
  • 174
  • 1
    Is it safe to do on mounted fs? On lab environment (fresh ext4 on loop device) after kill_file for normal file and creating new file i got: `touch: cannot touch '/test/3': Input/output error` and `EXT4-fs error (device loop2): __ext4_new_inode:1111: comm touch: failed to insert inode 13: doubly allocated?` – bfuvx Jan 13 '18 at 18:11
  • The same result with `rm` via debugfs on mounted fs. So its unsafe to do. – bfuvx Jan 13 '18 at 18:18
  • @JoKin I am not a filesystem expert and thus cannot tell you if there is a way to get it really safe. I guess it is rather safe as long as the page cache does not contain the affected area. I have forgotten how to do that but IIRC you can somehow throw away the whole page cache. – Hauke Laging Jan 13 '18 at 18:18
  • @JoKin Because you asked "without unmounting and full checking fs" I assumed that `umount` alone (without `fsck`) would be OK for you. – Hauke Laging Jan 13 '18 at 18:19
1

In my case, it was because the filesystem was mounted as a cifs smb/samba share with these global options:

[global]
   vfs objects = catia fruit streams_xattr
   fruit:aapl = yes

That provides compatibility with Apple computers and their desire to create secondary metadata streams for all media types (e.g. mp4).

But the way it works is that it creates invisible dotfiles (e.g. .apple.mp4 for apple.mp4, which can't be removed by a remote system, and which can get out of sync if the local system deletes apple.mp4 but not the dotfile.

The solution is to go back to the local system, where the dotfiles are visible and can be rm'd.

Guillaume
  • 11
  • 1