3

Configuration : Elasticsearch Version = 5.0.1 OS = Amazon Linux AMI release 2014.03

I am using AWS m3.large instance. I have two elasticsearch nodes. For my data node(node-2) , When I started elasticsearch service with

sudo service elasticsearch start

I found this as the error log.

[2016-11-25T09:31:56,884][INFO ][o.e.b.BootstrapCheck     ] [node-2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2016-11-25T09:31:56,886][ERROR][o.e.b.Bootstrap          ] [node-2] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]

Then I edited file /usr/lib/systemd/system/elasticsearch.service and added line

LimitNPROC=2048

When I restarted service. I checked status:

sudo service elasticsearch status

It returned,

elasticsearch dead but subsys locked

In Log 

[2016-11-25T09:50:32,138][INFO ][o.e.b.BootstrapCheck     ] [node-2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2016-11-25T09:50:32,140][ERROR][o.e.b.Bootstrap          ] [node-2] node validation exception
bootstrap checks failed
max number of threads [1024] for user [elasticsearch] is too low, increase to at least [2048]

The output of 

[ec2-user@ip-address]$ ls -la /var/lock/subsys
total 8
drwxr-xr-x 2 root root 4096 Nov 25 09:50 .
drwxrwxr-x 5 root lock 4096 Nov 25 03:37 ..
-rw-r--r-- 1 root root    0 Nov  9 19:10 atd
-rw-r--r-- 1 root root    0 Nov  9 19:08 auditd
-rw-r--r-- 1 root root    0 Nov  9 19:08 blk-availability
-rw-r--r-- 1 root root    0 Nov  9 19:10 crond
-rw-r--r-- 1 root root    0 Nov 25 09:50 elasticsearch
-rw-r--r-- 1 root root    0 Nov  9 19:10 local
-rw-r--r-- 1 root root    0 Nov  9 19:07 lvm2-monitor
-rw-r--r-- 1 root root    0 Nov  9 19:08 messagebus
-rw-r--r-- 1 root root    0 Nov  9 19:08 netfs
-rw-r--r-- 1 root root    0 Nov  9 19:08 network
-rw-r--r-- 1 root root    0 Nov  9 19:34 ntpd
-rw-r--r-- 1 root root    0 Nov  9 19:09 ntpdate
-rw------- 1 root root    0 Nov  9 19:08 rsyslog
-rw-r--r-- 1 root root    0 Nov 18 05:07 salt-minion
-rw-r--r-- 1 root root    0 Nov  9 19:10 sendmail
-rw-r--r-- 1 root root    0 Nov  9 19:10 sm-client
-rw-r--r-- 1 root root    0 Nov  9 19:09 sshd

Am I missing something ?

Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
Jakir patel
  • 31
  • 1
  • 6

2 Answers2

0

You can´t start elasticsearch because a dangling lock file is present in the system.

It is sufficient to delete it with:

sudo rm /var/lock/subsys/elasticsearch

Then you will be able to start elasticsearch again.

If this happens regularly, I would advise investigating the reasons behind it.

Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
  • Thank you for your response. I tried what you suggested and got following output. ` [ec2-user@subsys]$ sudo rm elasticsearch [ec2-user@subsys]$ sudo service elasticsearch status elasticsearch is stopped [ec2-user@subsys]$ sudo service elasticsearch start Starting elasticsearch: [ OK ] [ec2-user@subsys]$ sudo service elasticsearch status elasticsearch (pid 32500) is running... [ec2-user@]$ sudo service elasticsearch status elasticsearch dead but subsys locked ` – Jakir patel Nov 25 '16 at 13:49
  • Apologize for bad formatting @RuiFRiberio. As per your answer I got same output as previous comment. – Jakir patel Nov 25 '16 at 13:55
  • do the * rm*. again, then `ulimit -u 2048` . as root, then start it. – Rui F Ribeiro Nov 25 '16 at 14:05
  • I got this output @RuiFRiberio `[root@subsys]# rm elasticsearch rm: remove regular empty file ‘elasticsearch’? [root@subsys]# ulimit -u 2048 [root@]# sudo service elasticsearch status elasticsearch dead but subsys locked [root@subsys]# sudo service elasticsearch stop [root@subsys]# sudo service elasticsearch start Starting elasticsearch: [ OK ] [root@subsys]# sudo service elasticsearch status elasticsearch dead but subsys locked [root@subsys]#` – Jakir patel Nov 26 '16 at 08:04
  • 1
    I upgraded my Amazon Linux(2014.03) to Amazon linux (2016.09). Now elasticsearch is working fine. – Jakir patel Nov 30 '16 at 10:54
  • I suggest you add an answer to your own question, then I will delete this answer. – Rui F Ribeiro Nov 30 '16 at 11:58
0

I upgraded my Amazon Linux(2014.03) to Amazon Linux(2016.09). Now elasticsearch is working fine.

Jakir patel
  • 31
  • 1
  • 6