Unix and Linux Stack Exchange
Unix and Linux Stack Exchange

Questions tagged [logstash]

Logstash collects data from many types of logs, filters it, and outputs the result using many plugins. It is often associated with Elasticsearch and Kibana which have their own tags.

Logstash is an open source real-time data processing tool that accepts data from many different sources; it filters, transforms or normalises it in many ways, and outputs it to different "stashes", which can be analysis tools such as , or just senders of email alerts.

It has several hundred plugins that enable it to handle such diverse data as remote Apache logs sent over tcp sockets, or metrics from tools like Ganglia and collectd, or even IoT sensor values. is used to visualize the results.

See the documentation on the elastic.co site which groups these products together.

22 questions
2
votes
1 answer

what is GREEDYDATA in elasticsearch

reading the conf files of logstash i found in filter conf grok { match => { "message" => "Put\s*command\s*:\s+%{GREEDYDATA:command}" } } How does this filter work , i tried to search for GREEDYDATA but i couldn't understand
I'm V
  • 43
  • 1
  • 4
2
votes
0 answers

Rsyslog doesn't send logs to logstash server on port

I have two rsyslog-client and rsyslog-server. When I tried to "logger -p local1.notice SOMETHING" I don't see any information on server from one client, but from another I see logstash output. I have copied rsyslog.conf, it's identical on these two…
Vladimir Fomin
  • 177
  • 1
  • 1
  • 8
2
votes
1 answer

Tile/Greographic Map in Kibana not working

I am trying to create a geographical map of my data in Kibana 5.01, and it does not work. The fact is that I do not even have the geoip.field that is required in the menu. I am sending data from IntelMQ, that is processed by logstash to get into…
Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
1
vote
1 answer

How to make bash script invoking logstash return prompt

For various reasons, I am not running logstash (7.10.1) as a service, but rather invoking it on-demand, in a bash script: /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/my_ls.conf & echo "" echo…
datsb
  • 183
  • 1
  • 1
  • 10
1
vote
0 answers

Nginx in UDP load balancing continue to send to an unreachable upstream server

I use Nginx to load balance traffic coming from udp syslog sources to logstash Configuration : user nginx; worker_processes auto; error_log /var/log/nginx/error.log; pid /run/nginx.pid; worker_rlimit_nofile 1000000; # Load dynamic modules. See…
Atreiide
  • 11
  • 1
1
vote
1 answer

rsyslog error in logstash

I have installed ELK and tried to configure rsyslog server with logstash but i am getting lot of \ while executing curl -XGET 'http://172.22.63.61:9200/logstash-*/_search?q=*&pretty' Rsyslog servr rsyslog-8.24.0-38.el7.x86_64er config rsyslog…
MOBIN TM
  • 11
  • 3
1
vote
1 answer

Logstash Unrecognized service Amazon Linux

I have been following this tutorial to install ELK stack in a remote server which runs on Amazon Linux. https://www.aytech.ca/blog/setup-elk-stack-amazon-linux/ I was able to install Elasticsearch and then to start it as a service.Then I installed…
Sandun
  • 115
  • 1
  • 1
  • 9
1
vote
0 answers

logstash: Trying to extract substrings from path

I'm trying to extract substrings from my path field in my logstash config. The 'path' field looks like this: /storage/logs/deployment/servers/hostname.example.com/server.log Inside a filter section I have this: ruby { code =>…
Aditya K
  • 1,950
  • 5
  • 18
  • 33
1
vote
1 answer

logstash - take 2 - filter to send messages from IntelMQ/python/redis to ELK

Following up on the heels of this question, https://stackoverflow.com/questions/40768603/logstash-trying-to-make-sense-of-strings-passed-by-intelmq-in-elasticsearch I am trying to create a refine/create a filter to receive messages from logstash to…
Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
1
vote
1 answer

geoip logstash filter not working in Debian

I am trying to setup the geoip filter for logstashin Debian Stretch, as per this question I installed the geoip-database package, however logstash logs complain of wrong database format. What to do?
Rui F Ribeiro
  • 55,929
  • 26
  • 146
  • 227
1
vote
2 answers

logstash regex match in if condition

In logstash filtering, I have multiple tags setup based upon different error conditions and all the tags has a prefix, something like "abc:" In the output, I want to send email based upon just "abc:*" exists in tags. I haven't come across such…
sudobash
  • 61
  • 1
  • 6
0
votes
1 answer

keytool error: java.io.IOException: Invalid keystore format

I have a 3-node ELK stack (Elasticsearch v7.17). After a reboot, the Kibana web interface reports an error "Kibana server is not ready yet". The SSL certs were expired, so I re-created them (for the ELK CA, all 3 nodes, Kibana, and Logstash).…
dr_
  • 28,763
  • 21
  • 89
  • 133
0
votes
0 answers

Unable to send logs from rsyslog to logstash and elasticsearch

I am using ubuntu and I installed the ELK stack version 8.5 on the same machine. I did the necessary configurations for each of the services(logstash, elasticsearch, kibana) and I equally configured rsyslog to send logs to logstash(defining an index…
Ngouaba Rosalie
  • 31
  • 5
0
votes
1 answer

how to enable debug logs in stdout for logstash?

I'm struggling as newbie in logstash, below is some info of my env Logstash Version: logstash-7.16.2-1.x86_64 java Version: openjdk version "11.0.13" 2021-10-19 LTS Logstash Conf input { stdin { } } output { stdout { debug => true …
Sollosa
  • 1,887
  • 4
  • 19
  • 32
0
votes
1 answer

prevent inode reuse

We are using Logstash to ingest our logs and we are facing some issues due inodes being reused. We tried all possible options on Logstash side so we are exploring the OS side. As far as I can see, if I create a file, drop it and later on I create a…
sickfear
  • 3
  • 1
1
2