4

Let's say I have a Linux system, either Debian, Ubuntu, or CentOS, where a user removed /lib, for example. Let's also say that I managed to boot this system with a livecd version of whatever OS it was running. Is it possible to run a command that will rebuild /lib on the damaged system?

For example, I would guess that something like this could work by re-installing all packages that the target system had in use.

I had a colleague encounter this situation, but the end result was a full reinstall. However, that got me thinking if a repair is possible, and perhaps quicker?

slm
  • 363,520
  • 117
  • 767
  • 871
cat pants
  • 753
  • 5
  • 23
  • 38

4 Answers4

4

In most cases this should be relatively easy to fix. You can generate a list of all installed packages using the /lib directory with:

dpkg -S /lib | awk -F': ' '{ print $1 }' | tr -d ,

If you reinstall these packages, then it should fix the issue:

apt-get --reinstall $(dpkg -S /lib | awk -F': ' '{ print $1 }' | tr -d ,)

This of course depends on the deleted directory not being critical for apt-get. Packages required for apt-get to work would have to be installed manually, perhaps even without dpkg. In this case the package could be extracted on another machine and the files copied into place, they would later be re-installed with apt anyway so this shouldn't cause any other problems.

Looking at my own list of packages putting files in /lib, I see that major ones are the kernel (all modules go to /lib/modules) and libc. These are two that would almost certainly have to be reinstalled for the system to be bootable. One way to do this would be to install from a live Debian based system. If you download the necessary packages and mount the root of the target system, you can add the --root option to the dpkg there to install, eg:

dpkg --root=/path/to/target/root -i package1.deb package2.deb ...

The --root option could also be used with first dpkg command if necessary. Of course with enough patience and skill, it should be possible to repair online. If the dpkg command fails to work, you can always attempt to look up package dependencies directly in /var/lib/dpkg/info and search for packages putting files in /lib via the .list files in /var/lib/dpkg/info.

To give an idea of what can be achieved, here is a blog about someone who managed to recover a running Gentoo system where all the packages had been uninstalled. It makes for an interesting read, plus some of the techniques used (or the one suggested in comments) could be applied in this kind of situation - http://fakeguido.blogspot.co.uk/2010/08/rescuing-hosed-system-using-only-bash.html

Graeme
  • 33,607
  • 8
  • 85
  • 110
  • Will these applications work w/o `/lib` not being present? – slm Apr 16 '14 at 22:30
  • @slm, you would have to add all the necessary files for them to work. – Graeme Apr 16 '14 at 22:34
  • Yeah you're thinking of manually adding them as you encounter them to prop the system up enough that they're running. Then proceed with the rest of the recovery once that's in place. I'd buy that approach. – slm Apr 16 '14 at 22:36
  • Assuming it works you might need to enlist `ldd ` to determine missing libraries connecting the dots to which package they're a part of, essentially working backwards. – slm Apr 16 '14 at 22:37
  • @slm or just inspect dependencies directly in `/var/lib/dpkg/available`. – Graeme Apr 16 '14 at 22:48
  • I think it's the "--root" option for dpkg that makes this all possible. Booting into a live CD seems like the best strategy. Thanks! – cat pants Apr 18 '14 at 19:36
1

I think in most of the cases without a functioning /lib directory many of the applications you'd need to do a repair job will no longer be present. I would bite the bullet and just do a re-install.

If you're bent on trying anyway then here's what to do for RedHat based distros.

For Redhat based distros such as CentOS/Fedora/RHEL you can use RPM to verify and repair some aspects of the installed packages.

verify all pkgs

% rpm -qVav
.........    /usr/bin/rdesktop
.........    /usr/share/doc/rdesktop-1.6.0
.........  d /usr/share/doc/rdesktop-1.6.0/AUTHORS
.........  d /usr/share/doc/rdesktop-1.6.0/COPYING
.........  d /usr/share/doc/rdesktop-1.6.0/ChangeLog
.........  d /usr/share/doc/rdesktop-1.6.0/HACKING
.........  d /usr/share/doc/rdesktop-1.6.0/README
.........  d /usr/share/doc/rdesktop-1.6.0/TODO
.........  d /usr/share/doc/rdesktop-1.6.0/ipv6.txt
.........  d /usr/share/doc/rdesktop-1.6.0/keymap-names.txt
.........  d /usr/share/doc/rdesktop-1.6.0/keymapping.txt
...
...

verify openssh

% rpm -qVv openssh
.........    /etc/ssh
..?......  c /etc/ssh/moduli
.........    /usr/bin/ssh-keygen
.........    /usr/libexec/openssh
.........    /usr/libexec/openssh/ssh-keysign
.........    /usr/share/doc/openssh-5.5p1
.........  d /usr/share/doc/openssh-5.5p1/CREDITS
.........  d /usr/share/doc/openssh-5.5p1/ChangeLog
.........  d /usr/share/doc/openssh-5.5p1/INSTALL
.........  d /usr/share/doc/openssh-5.5p1/LICENCE
.........  d /usr/share/doc/openssh-5.5p1/OVERVIEW
...
...

fix permissions and ownership

% rpm --setperms {packagename}
% rpm --setugids {packagename}

NOTE: See man rpm for more details on -V|--verify output.

See this article for more details: http://www.cyberciti.biz/tips/reset-rhel-centos-fedora-package-file-permission.html.

slm
  • 363,520
  • 117
  • 767
  • 871
1

As others have said, I would probably do a reinstall if that happened to me. However, if I was trying to do a repair, I'd first start by copying the /lib directory from a live install into my system. If all of /lib was deleted, then nothing would work at first, not even the package manager(s), so such a step might be necessary. Then I'd go into my package manager, and reinstall all the packages with files in /lib that were listed as being installed (see Graeme's answer for how to obtain such a list). Then as a final cleanup, I'd go in and delete all the /lib files copied from the live install that should not be there; i.e. did not correspond to files from an installed package.

Faheem Mitha
  • 34,649
  • 32
  • 119
  • 183
0

In terms of being quicker I think reinstall would be. As you would still need to pick system peace by peace, not to count the time it might take you to later debug some issues that could be caused by this, it would take you longer. I would suggest just rebuilding the machine as that will definitely bring system to a stable state.

ek9
  • 2,875
  • 3
  • 18
  • 27