3

I have an m1.small EC2 instance that is mostly just running Apache as a web server for several simple PHP web sites (that use RDS as a database). The server is constantly with a very high load average - around 8, and never below 5. This causes my web sites to be annoyingly slow, much more then I'd expect from a supposedly 2.6GHz CPU with 1.7GB RAM.

top and friends shows that other then ~50% "steal" time (AFAIU, time stolen by the hypervisor for other VMs on the same CPU), the rest is almost entirely "IRQ" time.

mpstat says that this time is spend on a very stable 191.59 interrupts/sec (i.e. that number almost doesn't change over time), and according to '/proc/interrupts' these are mostly spend on xen-percpu-virq timer0 and xen-dyn-event eth0.

What are these, and how can I get the EC2 instance to lower the load and spend some more time on my PHP sites?

Guss
  • 12,278
  • 2
  • 22
  • 39
  • What does output of `/cat /proc/interrupts` show? – slm Jun 07 '14 at 17:32
  • I'm a little suspicious that you're expecting too much from the m1.small.EC2 instances. http://blog.patrickmeenan.com/2012/05/ec2-performance.html. Also this: http://blog.carlmercier.com/2012/01/05/ec2-is-basically-one-big-ripoff/ – slm Jun 07 '14 at 18:20
  • `/proc/interrupts` just shows high counts for the items specified in the question, and low or zero for everything else. Regarding performance - I don't think I'm expected a lot - a celeron class CPU should be able to run a requested through a somewhat customized wordpress site in less then 16 seconds (which is my current performance). – Guss Jun 08 '14 at 06:37
  • 1
    This sounded similar to your issue: http://serverfault.com/questions/446173/cpu0-is-swamped-with-eth1-interrupts – slm Jun 08 '14 at 12:25
  • There's also this official thread in the AWS forums: http://serverfault.com/questions/446173/cpu0-is-swamped-with-eth1-interrupts. Is `irqbalance` running? – slm Jun 08 '14 at 12:27
  • @sim: the problems you are discussing are about SMP systems, while m1.small is a single core VM - so I don't see how IRQ balancing has any relevance. Thanks. – Guss Jun 08 '14 at 12:39
  • OK, just trying to help move this Q along. Is there any additional info you can give us about this? – slm Jun 08 '14 at 12:42
  • Nothing comes to mind, except maybe the actual output of `/proc/interrupts` and `top`, though I doubt that will give a lot of insight. I have a feeling the hardware is simply very loaded by other workflows and I can "fix" my problem by moving the VM to another host. – Guss Jun 08 '14 at 13:02
  • 1
    Actually, @sim, reading the serverfault question again, and after finding an AWS support thread with a similar issue, I think this is relevant. Thanks! – Guss Jun 09 '14 at 05:56

1 Answers1

1

After looking at it for a while, this seems like a "hardware issue" - most likely the physical server hosting the VM is under heavy load from other VMs, and the m1 type machines do not get guaranteed performance, so you get whatever CPU time is left on the host. Worse, if your VM is running on CPU core 0, you also get all the interrupt handling on the machine. See this Xen wiki article under "HVM VM's first (and possibly only) VCPU is fully utilised.". Also see this AWS support thread for other people with a similar interrupt problem.

The only solution at this point is to get your VM migrated to another physical host. This can sometimes be done very simply by stopping the VMs, waiting a few minutes and starting again - if EC2 has assigned another VM to your old host in the mean time, then you can get a different slot and the problem may be solved.

If that doesn't work, the best bet is to create a new image from your VM, and start it on another availability zone. This will cause IPs to be replaced and may require updating security groups and firewall rules. Also make sure you stop your VM before you create the image - to make sure that after the image is created and before the new VM is started, your old VM will not generate new data.

Guss
  • 12,278
  • 2
  • 22
  • 39
  • 2
    Please note that this answer is very old. Currently stopping a VM will always cause its IP to be lost and a new IP will need to be generated when it is started. Also the underlying technology has also changed quite a lot and this problem is unlikely to happen. – Guss Dec 10 '15 at 15:18