使用supervisor 管理logstash

supervisor和logstash此处不做介绍，直接进行操作配置

环境准备

需要创建一个普通用户，如jianxp以下均使用该用户部署服务，设用户目录为 /home/jianxp，安装目录为 /opt

准备部署包 jdk/supervisor/logstash 上传至服务器，默认在 /tools 目录下，解压到/opt目录下。

JDK安装

若 已安装 jdk1.8.0，则只需要建立已有JDK的软连接，可跳过后面的步骤：
```
mkdir -p /opt/javaln -sf ${JAVA_HOME} /opt/java/jdk
```

解压安装包：

tar zxf jdk-8u191-linux-x64.tar.gz -C /optln -s /opt/jdk1.8.0_191 /opt/jdk

添加环境变量：

编辑文件 vi ~/.bash_profile，追加如下环境变量：

export JAVA_HOME=/opt/jdkexport PATH=$JAVA_HOME/bin:$PATHexport CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

执行命令 source ~/.bash_profile 使环境变量生效。
执行命令 java -version，若输出 java version "1.8.0_191"，则说明jdk安装成功

Supervisor安装

离线安装supervisor，解压安装包 tar -zxf /tools/supervisor-3.3.4.tar -C /opt/，依次执行下列步骤

cd /opt/supervisor-3.3.4tar -zxf setuptools-32.3.1.tar.gz && cd setuptools-32.3.1python setup.py install --usercd .. && tar -zxf meld3-1.0.2.tar.gz && cd meld3-1.0.2python setup.py install --usercd .. && tar -zxf supervisor-3.3.4.tar.gz && cd supervisor-3.3.4python setup.py install --user

编辑supervisor配置文件

mkdir -p /opt/supervisor && cd /opt/supervisor && vi supervisord.conf

supervisord.conf 文件内容如下

[unix_http_server]file=/opt/supervisor/supervisor.sock; (the path to the socket file)chmod=0700; sockef file mode (default 0700)[supervisord]logfile=/opt/supervisor/supervisord.log;  (main log file;default $CWD/supervisord.log)pidfile=/opt/supervisor/supervisord.pid; (supervisord pidfile;default supervisord.pid)childlogdir=/opt/supervisor; ('AUTO' child log dir, default $TEMP); the below section must remain in the config file for RPC; (supervisorctl/web interface) to work, additional interfaces may be; added by defining them in separate rpcinterface: sections[rpcinterface:supervisor]supervisor.rpcinterface_factory=supervisor.rpcinterface:make_main_rpcinterface[supervisorctl]serverurl=unix:///opt/supervisor/supervisor.sock; use a unix:// URL  for a unix socket; The [include] section can just contain the "files" setting.  This; setting can list multiple files (separated by whitespace or; newlines).  It can also contain wildcards.  The filenames are; interpreted as relative to this file.  Included files *cannot*; include files themselves.[include]files = /opt/supervisor/conf.d/*.conf

创建配置文件里涉及到的目录
- 日志目录
mkdir -p /opt/supervisor && mkdir -p /opt/supervisor
- 管理进程的配置文件目录：
mkdir -p /opt/supervisor/conf.d

建立软连接

mkdir -p ~/.local/etcln -s /opt/supervisor/supervisord.conf ~/.local/etc/

至此，supervisor安装完毕
- 执行 supervisod 即可启动supervisor
- 执行 supervisorctl status，无报错则说明启动成功。
logstash 安装

解压安装包，添加软连接

tar -zxf /tools/logstash-6.8.9-fix.tar.gz -C /opt/ln -s /opt/logstash-6.8.9 /opt/logstash

配置logstash用supervisor管理（logstash的所有配置均在supervisor管理的配置文件中完成）

编辑配置文件，执行 cd /opt/supervisor/conf.d && vi logstash.conf

; logstash config file[program:logstash]command=/opt/logstash-6.8.9/bin/logstash directory=/opt/logstash-6.8.9/user=jianxpnumprocs=1stdout_logfile=/opt/logstash/logs/stderr.logstdout_logfile_maxbytes=64MBstdout_logfile_backups=1redirect_stderr=trueautostart=trueautorestart=truestartsecs=1stopwaitsecs=1killasgroup=truepriority=1说明command=/opt/logstash-6.8.9/bin/logstash   这个没有加-f等参数，默认读config/pipelines.yaml文件，此文件也是后续要配置的。

在/opt/logstash-6.8.9/config/conf.d 创建两个config文件，作为logstash的pipeline配置

vim config/conf.d/test.confinput{file{    path =>"/opt/logstash_test/test.log"    start_position=>"beginning" }}output { file {    path =>"/opt/logstash_test/output.log"} }vim config/conf.d/file.confinput{file{    path =>"/opt/logstash_test/test.log"    start_position=>"beginning"}}output { file {    path =>"/opt/logstash_test/output33.log"} }

上面两个pipeline时比较简单的，以文件输入和和文件输出为例子，两个pipeline都是以文件test.log为共同输入，然后输出到不同的文件中。

 编辑config/pipelines.yaml 文件，添加如下内容 - pipeline.id: file   pipeline.workers: 1   pipeline.batch.size: 1   path.config: "/opt/logstash-6.8.9/config/conf.d/file.conf" - pipeline.id: test   pipeline.workers: 1   pipeline.batch.size: 1   path.config: "/opt/logstash-6.8.9/config/conf.d/test.conf"

上面的logstash的pipeline配置完成后，并且supervisor目录下的logstash.conf也配置完成，既可以使用supervisor启动logstash

supervisorctl updatesupervisorctl status

在/opt/logstash-6.8.9/logs 目录下，可以查看logstash输出的日志，通过日志可以看到，Pipelines running count=2，表示启动两个pipeline。

可以开启另外一个shell窗口，向test.log文件中增加内容，log中也会输出Opening file 等信息。

[2021-03-10T16:18:04,953][INFO ][logstash.pipeline        ] Pipeline has terminated {:pipeline_id=>"test", :thread=>"#<Thread:0x24bd4c10 run>"}[2021-03-10T16:18:04,958][INFO ][logstash.runner          ] Logstash shut down.[2021-03-10T16:18:42,012][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.8.9"}[2021-03-10T16:18:47,360][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"file", "pipeline.workers"=>1, "pipeline.batch.size"=>1, "pipeline.batch.delay"=>50}[2021-03-10T16:18:47,721][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"test", "pipeline.workers"=>1, "pipeline.batch.size"=>1, "pipeline.batch.delay"=>50}[2021-03-10T16:18:47,940][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/opt/logstash-6.8.9/data/plugins/inputs/file/.sincedb_513b5c3d29a45c373b60caa3e09938d7", :path=>["/opt/logstash_test/test.log"]}[2021-03-10T16:18:47,940][INFO ][logstash.inputs.file     ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"/opt/logstash-6.8.9/data/plugins/inputs/file/.sincedb_513b5c3d29a45c373b60caa3e09938d7", :path=>["/opt/logstash_test/test.log"]}[2021-03-10T16:18:47,992][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"file", :thread=>"#<Thread:0x5c0e2cd6 run>"}[2021-03-10T16:18:47,993][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"test", :thread=>"#<Thread:0x6626055f run>"}[2021-03-10T16:18:48,074][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections[2021-03-10T16:18:48,074][INFO ][filewatch.observingtail  ] START, creating Discoverer, Watch with file and sincedb collections[2021-03-10T16:18:48,108][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:file, :test], :non_running_pipelines=>[]}[2021-03-10T16:18:48,531][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}[2021-03-10T16:19:11,774][INFO ][logstash.outputs.file    ] Opening file {:path=>"/opt/logstash_test/output.log"}[2021-03-10T16:19:11,776][INFO ][logstash.outputs.file    ] Opening file {:path=>"/opt/logstash_test/output33.log"}

查看输出的文件

[jianxp@localhost logstash_test]$ ll总用量 16-rw-rw-r--. 1 jianxp jianxp  335 3月  10 15:23 aa-rw-r--r--. 1 jianxp jianxp 3338 3月  10 16:19 output33.log-rw-r--r--. 1 jianxp jianxp 3338 3月  10 16:19 output.log-rw-rw-r--. 1 jianxp jianxp 1340 3月  10 16:19 test.log

每一份赞赏源于懂得

赞赏

0人进行了赞赏支持

使用supervisor 管理logstash