thinkphp5.1日志收集

php安装

  docker run -d --name laravel -p 8080:80 -v /data/php/thinkphp5:/var/www/html/app laraedit/laraedit

/data/php/thinkphp5是web目录

FILEBEAT安装

  filebeat.prospectors:
- input_type: log
  paths:
    - /var/www/html/app/runtime/log/201901/*.log
  multiline.pattern: '\-{63}'
  multiline.negate: true
  multiline.match: after
output.redis:
   #ip
  hosts: ["172.18.194.221"]
  port: 6379
  # password: ""
   #key
  key: "log"
  db: 0
  timeout: 5
  template.enabled: true
  template.path: "filebeat.template.json"
  template.overwrite: false

paths是日志位置

multiline.pattern匹配正则

elasticsearch安装

  docker run -e ES_JAVA_OPTS="-Xms256m -Xmx256m" -d -p 9200:9200 -p 9300:9300 --name elasticsearch elasticsearch:6.5.4

9200为elasticsearch的api接口

logstash安装

  docker run -d -v /data/logstash/logstash.conf:/usr/share/logstash/pipeline/logstash.conf -v /data/logstash/logstash.yml:/etc/logstash/logstash.yml --name logstash logstash

/data/logstash/logstash.conf为logstash的配置文件

/data/logstash/logstash.yml为系统配置文件

  input {
  redis {
      port => "6379"
      host => "172.18.194.221"
      data_type => "list"
      type => "log"
      key => "log"
  }
}

filter {
  grok {
    match => { "message" => "\-{63}\n\[ %{TIMESTAMP_ISO8601:timestamp:ts-rfc3339} \] %{IPV4:ip} %{WORD:methods} %{GREEDYDATA:url}\n%{GREEDYDATA:msg}" }
    remove_field => ["message"]
    }
}

output{
elasticsearch {
hosts => ["172.18.194.221:9200"]
index => "log"
}

input为采集程序缓冲用的redis队列

filter为正则匹配日志方式

output为输出到elasticsearch里

  pipeline:
batch:
  size: 125
  delay: 50
path:
data: /app/data/logstash

为logstash系统配置

grafana安装

  docker run -d --name=grafana -p 3000:3000 grafana/grafana

3000端口为grafana的web端口

redis安装

  docker run --name redis -p 6379:6379 -d redis