3

What is a secure update plan for web-facing servers when the person tasked with administration cannot allocate significant time to testing each updated package for breakage? Of course I want to update the LAMP stack regularly for security reasons, but I don't need the latest Java patch breaking Solr, or the latest PHP patch breaking a script in a non-obvious way. I do know to be wary of PHP point releases but with Java I am less confident. Right now our servers are Debian-based (Ubuntu Server LTS) but I am considering switching to Red-hat based (CentOS) because of Yum's rollback feature.

Are my concerns unwarranted? Note that this is a small firm, we just hired our fourth employee. So in addition to my duties as a software developer, I am tasked with maintain a growing list of hardware including public webservers. No sensitive data is on the webservers, the worst that could happen from an exploit is that someone would deface our website, which is also our product (SaaS through a public API).

dotancohen
  • 15,494
  • 26
  • 80
  • 116

2 Answers2

3

Select a Linux distribution "for enterprise use", install just the packages needed for its job (extra packages mean extra vulnerable surface), don't install any "unofficial packages" unless strictly required (and then only carefully considered ones for stability and upstream responsiveness to security problems, and track record/commitment to not trampling on the distribution's packages), configure with care (local firewall rules, any local configuration). Create local accounts as needed, give them secure passwords. Use root as little as possible on the machine. Set up passwordless SSH for selected users. Don't follow the many suggestions on the 'net to "just turn off SELinux", SELinux is able to provide much added security. Work with it, if a package doesn't work with SElinux, it is broken and shouldn't be used.

Subscribe to the distribution's package update announcements, update at least the security updates ASAP (not "once a month", at least check daily). Look at LWN's security page regularly.

Whatever services the machine will provide, select the packages to do so with care among what your distribution offers. It will probably be that the overall best is harder to use... some ideas on selecting/evaluating software are here.

Look around for "best practices" for the services to be offered, in particular configuration suggestions for what you install. Check if the online documentation is complete, clear, and searchable. See if there is a way to report bugs, check a random selection of bugs to see how responsive they are. Look for online question and answer sites, FAQs, mailing lists (if you run into trouble, you want to know who you can ask).

Design a comprehensive backup scheme, sooner or later you will have to fall back on it. Document the installation (or even better, set up automated installation, and keep it up to date), something untimely might happen to the machine. Make sure you have "old" backups too (some miscreant might take over the machine, and you find out a few months later...).

Consider hiring a few geek specialized in what you want to set up, just make sure to (a) keep them close at hand, and (b) you understand enough of the setup to be able to muddle through for most cases, and (c) make sure you pass the bus test (i.e., have others around that can take over in case you get run over by a bus).

Set up a machine with the same operating system for personal use, it is very useful in case of crisis to be intimately familiar with the ailing system (or even have a spare on hand ;-). This will be a part-time job, as in "it takes only 95% of my time", unless you take the time to organize the tasks so to minimize the unexepected (or you end up being the fire brigade being called 24/7).

Personally I'd use either Red Hat Enterprise Linux or a clone like CentOS, perhaps supplemented with EPEL. But that's just me, Red Hat user from around '95, lately Fedora fan.

vonbrand
  • 18,156
  • 2
  • 37
  • 59
  • 2
    To make it clear, by password-less SSH accounts we understand "disable password logins" **not** "use empty passwords". IOW use key-based authentication. And disable root login, of course. The `/etc/ssh/sshd` directives are: `PasswordAuthentication no` and `PermitRootLogin no`. – peterph Mar 19 '13 at 17:23
  • Thank you. For the latest MySQL and PHP I am considering the Remy repo. From what I've read EPEL makes no claim to support CentOS, can you elaborate on that? Thanks! – dotancohen Mar 19 '13 at 18:35
  • Don'r get fooled into "I need latest , and that is available only at ", the distribution does keep up with bugs and security issues, even while maintaining the base version fixed (see [here](http://unix.stackexchange.com/questions/62202/why-does-red-hat-linux-use-such-an-old-kernel) for a discussion). And EPEL does support RHEL, ergo it also supports clones of RHEL such as CentOS. I don't know if they state so explicitly (they might not do so because of the ties with Red Hat), but they do care for CentOS and Scientific Linux (and even Oracle Linux ;-). – vonbrand Mar 19 '13 at 18:42
  • CentOS is basically RedHat (binary compatible), so there should be no issues, AFAIU. – peterph Mar 19 '13 at 21:19
  • @peterph, one of CentOS explicit guidelines is to be binary compatible with RHEL always. – vonbrand Mar 19 '13 at 21:21
  • @vonbrand yes, that's exactly what I meant. IIRC their packages are just repackaged RH's (mostly with stripped branding). – peterph Mar 19 '13 at 21:28
  • 1
    Stay away from Remy on production machines. EPEL and the baseline set of CentOS repos is fine. Cutting-edge software in general has no place on production rigs, since cutting edges by definition cut and lead to bleeding and frustration. While the devs want to use the latest gadgets, you have to be the anchor of stability... – Deer Hunter Mar 19 '13 at 21:46
  • ...and yes, get rid of ready-made LAMP stacks. – Deer Hunter Mar 19 '13 at 21:48
  • @DeerHunter: Thanks. I do make heavy use of PHP >= 5.3 features, which is why I preferred Remy. Maybe I'll look into compiling it myself. – dotancohen Mar 21 '13 at 06:11
  • Why should we get rid of ready-made LAMP stacks? I would assume that the distro's package maintainers are more skilled in the art than myself. – dotancohen Mar 21 '13 at 06:12
1

First of all, strip the system to the bare necessities, remove everything that you don't need. If you don't need Java, get rid of it, if you don't need Perl, get rid of it... Then the obvious thing is to update everything that is left. Updates on fixed-versions distributions like Debian/Ubuntu, RH/CentOS/Fedora, SLES/openSUSE are unlikely to break anything, unless you are misusing the packages in question in some weird way.

If you are worried that a PHP update will break the website which, if I understand correctly, is your product, fix the product (if it's not the product, fix the script anyway). Otherwise it's just asking for troubles.

Defacing your web is definitely not the worst thing that can happen - once the system is hijacked, it can (and likely will) be misused in much subtler and more dangerous ways than just putting offending stuff on public display.

peterph
  • 30,520
  • 2
  • 69
  • 75