logstash 收集日志

一、logstash使用學習

不難理解,我們的日志通常都是在日志文件中存儲的,所以,當我們在使用INPUT插件時,收集日志,需要使用file模塊,從文件中讀取日志的內容,那么接下來講解的是,將日志內容輸出到另一個文件中,如此一來,我們可以將日志文件同意目錄,方便查找。

注意:Logstash與其他服務不同,收集日志的配置文件需要我們根據實際情況自己去寫。
前提:需要Logstash對被收集的日志文件有讀的,并且對要寫入的文件,有寫入的權限。

1.logstash配置文件

[root@web01 ~]# vim /etc/logstash/logstash.ymlpath.config: /etc/logstash/conf.d

2.logstash收集單個日志到文件

1)配置

[root@web01 ~]# cd /etc/logstash/conf.d/[root@web01 /etc/logstash/conf.d]# vim message_file.confinput {    file {        path => "/var/log/messages"        start_position => "beginning"    }}output {    file {        path => "/tmp/messages_%{+YYYY-MM-dd}"    }}[root@web01 /etc/logstash/conf.d]# vim message_file.conf#輸入插件input {#文件模塊    file {#日志類型        type => "message-log"#日志路徑        path => "/var/log/messages"#第一次收集日志從頭開始        start_position => "beginning"  }}#輸出插件output {#文件模塊    file {#輸出路徑        path => "/tmp/message_%{+yyyy.MM.dd}.log"    }}

2)啟動

#檢測語法[root@web01 ~]# logstash -f /etc/logstash/conf.d/message_file.conf -t#啟動[root@web01 ~]# logstash -f /etc/logstash/conf.d/message_file.conf &

3)測試日志收集

#實時監控收集到的日志[root@web01 ~]# tail -f /tmp/messages_2020-12-04#手動添加一臺日志[root@web01 ~]# echo 111 >> /var/log/messages

3.logstash收集單個日志到ES

1)配置

[root@web01 ~]# vim /etc/logstash/conf.d/message_es.conf input {  file {    path => "/var/log/messages"    start_position => "beginning"  }}output {  elasticsearch {    hosts => ["10.0.0.71:9200"]    index => "message_%{+YYYY-MM-dd}"  }}

2)啟動

[root@web01 ~]# logstash -f /etc/logstash/conf.d/message_es.conf &[2] 82713

4.logstash啟動多實例

logstash收集日志時使用多實例方式啟動,不是使用system管理啟動,但是啟動多實例會報錯,怎么處理?

1)配置收集/var/log/secure日志

[root@web01 ~]# vim /etc/logstash/conf.d/secure_es.conf input {  file {    path => "/var/log/secure"    start_position => "beginning"  }}output {  elasticsearch {    hosts => ["10.0.0.71:9200"]    index => "secure_%{+YYYY-MM-dd}"  }}

2)啟動多實例

logstash只啟動一個不需要數據目錄,如果想要啟動多個進程,需要每個進程指定不同的數據目錄,需要加 --path.data參數,然后可以啟動多實例1.創建數據目錄[root@web01 ~]# mkdir /data/logstash/messages_es -p[root@web01 ~]# mkdir /data/logstash/secure_es -p[root@web01 ~]# chown -R logstash.logstash /data/logstash/2.分別指定數據目錄再啟動兩個進程[root@web01 ~]# logstash -f /etc/logstash/conf.d/message_es.conf --path.data=/data/logstash/messages_es &[root@web01 ~]# logstash -f /etc/logstash/conf.d/secure_es.conf --path.data=/data/logstash/secure_es &

5.logstash收集多個日志到文件

1)配置

[root@web01 ~]# vim /etc/logstash/conf.d/morefile_file.conf#輸入的插件input {  #文件模塊  file {    #收集文件的路徑    path => "/var/log/messages"#第一次收集從頭收集    start_position => "beginning"#收集日志間隔時間3秒    stat_interval => "3"  }  #第二個文件模塊  file {    #第二個收集日志的路徑    path => "/var/log/secure"  }}#輸出插件output {  #輸出時的文件模塊  file {    #輸出的文件路徑    path => "/tmp/morefile.txt"  }}

2)啟動

[root@web01 ~]# logstash -f /etc/logstash/conf.d/morefile_file.conf &

3)驗證

[root@web01 ~]# tail -f /tmp/morefile.txt#手動添加文件[root@web01 ~]# echo 111 >> /var/log/messages[root@web01 ~]# echo 2222 >> /var/log/secure

6.logstash收集多個日志到ES

1)配置

[root@web01 ~]# vim /etc/logstash/conf.d/morefile_es.conf input {  file {    path => "/var/log/messages"  }  file {    path => "/var/log/secure"  }}output {  elasticsearch {    hosts => ["10.0.0.71:9200"]    index => "morefile_%{+YYYY-MM-dd}"  }}

2)啟動

[root@web01 ~]# logstash -f /etc/logstash/conf.d/morefile_es.conf &

3)驗證

7.收集多個日志到多個索引

1)方法一:

#配置[root@web01 ~]# cat /etc/logstash/conf.d/morefile_es.confinput {  file {    type => "messages_log"    path => "/var/log/messages"  }  file {    type => "secure_log"    path => "/var/log/secure"  }}output {  if [type] == "messages_log" {    elasticsearch {      hosts => ["10.0.0.71:9200"]      index => "messages_log_%{+YYYY-MM-dd}"    }  }  if [type] == "secure_log" {    elasticsearch {      hosts => ["10.0.0.71:9200"]      index => "secure_log_%{+YYYY-MM-dd}"    }  }}#啟動[root@web01 ~]# logstash -f /etc/logstash/conf.d/morefile_es.conf

2)方法二:

#配置[root@web01 ~]# cat /etc/logstash/conf.d/second_morefile_es.confinput {  file {    type => "messages_log"    path => "/var/log/messages"  }  file {    type => "secure_log"    path => "/var/log/secure"  }}output {  elasticsearch {    hosts => ["10.0.0.71:9200"]    index => "%{type}_%{+YYYY-MM-dd}"  }}#啟動[root@web01 ~]# logstash -f /etc/logstash/conf.d/second_morefile_es.conf

二、使用logstash收集nginx日志

1.修改nginx日志格式為json格式

[root@web01 ~]# cat /etc/nginx/nginx.conf ... ...http {    include       /etc/nginx/mime.types;    default_type  application/octet-stream;    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '                      '$status $body_bytes_sent "$http_referer" '                      '"$http_user_agent" "$http_x_forwarded_for"';                          log_format  json  '{"@timestamp":"$time_iso8601",'                      '"host":"$server_addr",'                      '"clientip":"$remote_addr",'                      '"size":$body_bytes_sent,'                      '"responsetime":$request_time,'                      '"upstreamtime":"$upstream_response_time",'                      '"upstreamhost":"$upstream_addr",'                      '"http_host":"$host",'                      '"url":"$uri",'                      '"referer":"$http_referer",'                      '"agent":"$http_user_agent",'                      '"status":"$status"}';    #access_log  /var/log/nginx/access.log  main;    access_log  /var/log/nginx/access.log  json;    sendfile        on;    client_max_body_size 100M;    keepalive_timeout  65;    include /etc/nginx/conf.d/*.conf;}

2.重啟nginx訪問查看日志

[root@web01 ~]# systemctl restart nginx[root@web01 ~]# tail -f /var/log/nginx/access.log{"@timestamp":"2020-12-04T17:39:22+08:00","host":"10.0.0.7","clientip":"10.0.0.1","size":0,"responsetime":0.000,"upstreamtime":"-","upstreamhost":"-","http_host":"10.0.0.7","url":"/index.html","referer":"-","agent":"Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.92 Safari/537.36","status":"304"}

3.配置logstash收集nginx日志

[root@web01 ~]# vim /etc/logstash/conf.d/nginx_log_es.confinput {  file {    path => "/var/log/nginx/access.log"    start_position => "end"    type => "access_log"  }}output {  elasticsearch {    hosts => ["10.0.0.71:9200"]    index => "nginx_access_log_%{+YYYY-MM-dd}"  }}

4.啟動并測試

[root@web01 ~]# logstash -f /etc/logstash/conf.d/nginx_log_es.conf

三、logstash收集tomcat日志

在企業中,我們看到tomcat日志遇到異常(exception)一條日志可能是幾行或者十幾行甚至幾十行,組成的,那么,我們需要將多行日志變成一行日志,來收集

1.tomcat日志收集方式

這里我們有幾種方式可以實現:1.將日志改成Json格式在企業中,想要將java日志改成json格式,并沒有那么容易。因為將日志改成Json格式,查看起來會很難受,有些開發人員不希望將日志格式改成Json的,所以,在改日志格式之前需要跟開發人員進行溝通,那么將tomcat日志格式改成Json格式也有兩種方式。1)開發自己更改,通過程序代碼,或者log4j2)運維修改tomcat的server配置文件2.通過logstash的mutiline模塊實現多行匹配

2.安裝tomcat

1)安裝java環境

2)安裝tomcat

1.上傳代碼包[root@web01 ~]# rz[root@web01 ~]# ll-rw-r--r--  1 root root  11026056 2020-12-04 18:04 apache-tomcat-9.0.30.tar.gz2.解壓tomcat包[root@web01 ~]# tar xf apache-tomcat-9.0.30.tar.gz3.將安裝包移動并改名[root@web01 ~]# mv apache-tomcat-9.0.30 /usr/local/tomcat-9.0.304.做軟連接[root@web01 ~]# ln -s /usr/local/tomcat-9.0.30 /usr/local/tomcat

3)配置站點

1.寫一個測試頁面到站點目錄下的index.html文件中[root@web01 ~]# echo 'TEST elk' > /usr/local/tomcat/webapps/ROOT/index.html2.啟動tomcat[root@web01 ~]# /usr/local/tomcat/bin/startup.sh3.檢測tomcat端口是否啟動[root@web01 ~]# netstat -lntup|grep 8080tcp        0      0 :::8080                     :::*           LISTEN      12569/java

4)訪問測試

http://10.0.0.7:8080/

3.配置logstash收集tomcat日志

1)配置

[root@web01 ~]# vim /etc/logstash/conf.d/tomcat_log_es.conf input {  file {    path => "/usr/local/tomcat/logs/localhost_access_log.*.txt"    start_position => "end"    type => "tomcat_log"  }}output {  elasticsearch {    hosts => ["10.0.0.71:9200"]    index => "tomcat_log_%{+YYYY-MM-dd}"  }}

2)啟動

[root@web01 ~]# logstash -f /etc/logstash/conf.d/tomcat_log_es.conf