构建ELK+Redis日志分析系统

2016-07-07

安装Logstash+ElasticSearch5+Kibana5

JDK

最新版elasticsearch-5.0.0-alpha4需要java-1.8,

首先下载jdk-1.8,下载地址:http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

解压安装:

wget --no-check-certificate --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie"  http://download.oracle.com/otn-pub/java/jdk/8u92-b14/jdk-8u92-linux-x64.tar.gz
tar -zxvf jdk-8u92-linux-x64.tar.gz
mkdir /usr/local/java
mv jdk1.8.0_92 /usr/local/java/

设置环境变量:

vim ~/.bashrc 追加:

export JAVA_HOME=/usr/local/java/jdk1.8.0_92
export JRE_HOME=${JAVA_HOME}/jre  
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib  
export PATH=${JAVA_HOME}/bin:$PATH

验证安装版本:

root@ubuntu:~# java -version
java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)

ElasticSearch5

ElasticSearch官方下载:https://www.elastic.co/downloads/elasticsearch

官方提供deb包,那就偷懒下 elasticsearch-5.0.0-alpha4.deb

安装:

dpkg -i elasticsearch-5.0.0-alpha4.deb

默认安装在/usr/share/elasticsearch 目录

启动:

cd /usr/share/elasticsearch/
mkdir config
cd bin/
./elasticsearch

错误:

如果报错:

Exception in thread "main" java.lang.RuntimeException: can not run elasticsearch as root

那么切换到普通用户

如果还报错:

Exception in thread "main" java.lang.IllegalStateException: Unable to access 'path.conf' (/usr/share/elasticsearch/config)

是因为访问权限不够:

chown malu /usr/share/elasticsearch -R

中文文档:http://learnes.net/

https://www.gitbook.com/book/looly/elasticsearch-the-definitive-guide-cn/details

Kibana5

Kibana官方下载: https://www.elastic.co/downloads/kibana

deb包下载: kibana-5.0.0-alpha4-amd64.deb

安装:

dpkg -i kibana-5.0.0-alpha4-amd64.deb

默认安装在/usr/share/kibana

启动:

cd /usr/share/kibana/bin
./kibana

中文文档:http://www.code123.cc/docs/kibana-logstash/

logstash5

官方下载:https://www.elastic.co/downloads/logstash

deb下载:logstash-5.0.0-alpha4.deb

安装:

dpkg -i logstash-5.0.0-alpha4.deb

默认安装在/usr/share/logstash

启动:

首先需要配置运行文件:

vim /etc/logstatsh/conf.d/first-pipeline.conf

input {
    file {
        path => "/var/log/nginx/access.log"
        start_position => beginning
        ignore_older => 0
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {
        hosts => [ "localhost:9200","server-2.malu.me:9200" ]
        user => "username"
        password => "pass"
        index => "%{module}-%{+YYYY.MM.dd}"
        document_type => "%{msgtype}"
    }
}

启动前可以用命令检查一下参数是否正确:

bin/logstash -f first-pipeline.conf --config.test_and_exit

然后执行:

/usr/share/logstash/bin/logstash

此时会读取logstash默认配置文件 /usr/share/logstash/config/logstash.yml

如果不存在,请从/etc/logstash/logstash.yml复制

logstash 2.3.4

下载安装环境:

wget https://download.elastic.co/logstash/logstash/packages/debian/logstash_2.3.4-1_all.deb
dpkg -i logstash_2.3.4-1_all.deb
apt-get update && apt-get install -y openjdk-7-jre-headless

vim /etc/logstash/conf.d/logstash.conf

input {
    file {
        path => "/var/log/apache/access.log"
        start_position => beginning
        ignore_older => 0
    }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {
         hosts => "es.malu.me:6585"
         user => "malu"
         password => "elk"
    }
}

启动:

/opt/logstash/bin/logstash -f /etc/logstash/conf.d/logstash.conf

离线日志采集

离线日志文件对应的默认的 $HOME/.sincedb_*文件删掉再用logstash采集

文档:

Logstash 最佳实践 http://udn.yyuap.com/doc/logstash-best-practice-cn/index.html

redis

可以用redis来实现分布式日志管理

参考这篇文章:http://git.malu.me/Redis客户端常用命令/

配置redis_index.conf文件,直接与elasticsearch通讯:

root@ubuntu:/etc/logstash/conf.d# cat redis_index.conf
input {
        redis {
                host => "localhost"
                port => 6379
                password => "redispassword"
                data_type => "list"
                key => "logstash:redis"
                type => "redis-input"
        }
}
filter {
    grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
}
output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
    }
}

配置redis_cli.conf用于读取日志:

root@ubuntu:/etc/logstash/conf.d# cat redis_cli.conf 
input {
    file {
        path => "/var/log/nginx/access.log"
        start_position => beginning
        ignore_older => 0
    }
}
output {
        redis {
                host => "localhost:6379"
                password => "redispassword"
                data_type => "list"
                key => "logstash:redis"
        }
}

然后启动:

./logstash -f /etc/logstash/conf.d/redis_index.conf
./logstash -f /etc/logstash/conf.d/redis_cli.conf --http.port 9601   #启动第二个进程会提示9600端口占用,指定其他端口就可以了

中文文档:ELKstack 中文指南

ElasticSearch5监听外网端口

一般不建议把ElasticSearch监听放外网,但本人在终端环境下,用curl实在不爽。

首先拷贝配置文件:

cp /etc/elasticsearch/elasticsearch.yml /usr/share/elasticsearch/config/elasticsearch.yml

vim /usr/share/elasticsearch/config/elasticsearch.yml

network.host: 192.168.1.222
http.port: 9200
discovery.zen.minimum_master_nodes: 1    #这里设为1, 设3的话会导致Kibana报错

如果直接启动ElasticSearch会报错:

initial heap size [33554432] not equal to maximum heap size [522190848]; this can cause resize pauses and prevents mlockall from locking the entire heap
max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]
please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster
max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

有4个错误,我们一个个来解决:

initial heap size [33554432] not equal to maximum heap size [522190848]; this can cause resize pauses and prevents mlockall from locking the entire heap

这个错误是关于jvm内存分配的问题,需要设置的jvm配置:

cp /etc/elasticsearch/jvm.options /usr/share/elasticsearch/config/jvm.options

vim /usr/share/elasticsearch/config/jvm.options

-Xmx2g 改成 -Xmx256m ,也就是heap size [268435456] /1024/1024的值

第二个错误:

max file descriptors [4096] for elasticsearch process likely too low, increase to at least [65536]

最大文件打开句柄数太少了,需要设置成65536:

cp /etc/security/limits.conf /etc/security/limits.conf.bak

vim /etc/security/limits.conf

malu hard nofile 65536
malu soft nofile 65536

其中malu是启动ElasticSearch的用户名

修改后重新登录malu用户,使用如下命令查看是否修改成功

[malu@localhost ~]$ ulimit -Hn 65536

第三个错误:

please set [discovery.zen.minimum_master_nodes] to a majority of the number of master eligible nodes in your cluster

配置文件elasticsearch.yml里设置:

discovery.zen.minimum_master_nodes: 1 

第四个错误:

max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]

最大线程数设置过小,需要root用户来设置:

sysctl -w vm.max_map_count=262144

查看:

[root@localhost ~]# sysctl -a|grep vm.max_map_count
vm.max_map_count = 262144

下次启动这个设置会丢失,我们把它写入配置文件:

vim /etc/sysctl.conf

vm.max_map_count=262144  #添加这一行

以上文件解决后,elasticsearch就可以正常启动了。不过由于修改了监听地址,Kibana与它的通讯地址也得对应修改:

vim /etc/kibana/kibana.yml

elasticsearch.url: "http://192.168.1.222:9200"  #添加这一行

如果logstash配置了Redis与之通讯,那么同样也得修改:

output {
    elasticsearch {
        hosts => [ "192.168.1.222:9200" ]
    }
}

文档:

elasticsearch 集群传输端口配置官方文档

ELKstack 中文指南 : https://www.gitbook.com/book/chenryn/kibana-guide-cn/details

Elasticsearch 权威指南(中文版)

官网 : https://www.elastic.co

国内几个ELK相关博客:

http://chenlinux.com/

http://xiaorui.cc/category/elasticsearch/

http://bigbo.github.io/category/technology.html

Other

高性能的PHP日志系统—SeasLog

使用 Grafana、collectd 和 InfluxDB 打造现代监控系统