13.安装logstash
1. 内存使用¶
# 当前内存：11.82G
# 启动logstash后内存：12.53G！
# 内存使用大概：0.71G！
2. 安装logstash¶
cd /home/ubuntu
ls
dpkg -i logstash*
root@4c16g:/home/ubuntu# dpkg -i logstash*
Selecting previously unselected package logstash.
(Reading database ... 194085 files and directories currently installed.)
Preparing to unpack logstash-7.16.2-amd64.deb ...
Unpacking logstash (1:7.16.2-1) ...
Setting up logstash (1:7.16.2-1) ...
Using bundled JDK: /usr/share/logstash/jdk
Using provided startup.options file: /etc/logstash/startup.options
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/pleaserun-0.0.32/lib/pleaserun/platform/base.rb:112: warning: constant ::Fixnum is deprecated
Successfully created system startup script for Logstash
3. 测试logstash¶
# 从上面可以看到，logstash的bin目录在： /usr/share/logstash
/usr/share/logstash/bin/logstash -e 'input { stdin {}} output{ stdout {}}'
# 当按回车后，出现如下输出就说明没问题！
{
"host" => "node-1",
"@version" => "1",
"@timestamp" => 2021-12-10T06:25:11.222Z,
"message" => ""
}
root@4c16g:/home/ubuntu# /usr/share/logstash/bin/logstash -e 'input { stdin {}} output{ stdout {}}'
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-01-01 21:36:01.972 [main] runner - Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[INFO ] 2022-01-01 21:36:02.060 [main] settings - Creating directory {:setting=>"path.queue", :path=>"/usr/share/logstash/data/queue"}
[INFO ] 2022-01-01 21:36:02.095 [main] settings - Creating directory {:setting=>"path.dead_letter_queue", :path=>"/usr/share/logstash/data/dead_letter_queue"}
[WARN ] 2022-01-01 21:36:02.873 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-01-01 21:36:02.945 [LogStash::Runner] agent - No persistent UUID file found. Generating new UUID {:uuid=>"51a3c371-5cf7-4edb-854f-0be3348482f7", :path=>"/usr/share/logstash/data/uuid"}
[INFO ] 2022-01-01 21:36:04.972 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2022-01-01 21:36:05.820 [Converge PipelineAction::Create<main>] Reflections - Reflections took 149 ms to scan 1 urls, producing 119 keys and 417 values
[WARN ] 2022-01-01 21:36:06.724 [Converge PipelineAction::Create<main>] line - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-01-01 21:36:06.749 [Converge PipelineAction::Create<main>] stdin - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-01-01 21:36:07.082 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["config string"], :thread=>"#<Thread:0x445bc62c run>"}
[INFO ] 2022-01-01 21:36:08.004 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>0.92}
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.jrubystdinchannel.StdinChannelLibrary$Reader (file:/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/jruby-stdin-channel-0.2.0-java/lib/jruby_stdin_channel/jruby_stdin_channel.jar) to field java.io.FilterInputStream.in
WARNING: Please consider reporting this to the maintainers of com.jrubystdinchannel.StdinChannelLibrary$Reader
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO ] 2022-01-01 21:36:08.102 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
The stdin plugin is now waiting for input:
[INFO ] 2022-01-01 21:36:08.180 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
{
"@timestamp" => 2022-01-01T13:36:08.218Z,
"@version" => "1",
"message" => "",
"host" => "4c16g"
}
{
"@timestamp" => 2022-01-01T13:36:08.233Z,
"@version" => "1",
"message" => "",
"host" => "4c16g"
}
{
"@timestamp" => 2022-01-01T13:36:08.232Z,
"@version" => "1",
"message" => "",
"host" => "4c16g"
}
4. 查看配置文件¶
#1.复制
cp /etc/logstash/logstash-sample.conf /etc/logstash/conf.d/logstash-to-elasticsearch.conf
#2.修改配置文件
vim /etc/logstash/conf.d/logstash-to-elasticsearch.conf
#1. 修改12行，将localhost改为10.0.8.5，es所在的主机（如果在同一台机器，不修改也可以！）
hosts => ["http://10.0.8.5:9200"]
#2. 添加18-20行，将输出打印到屏幕上！
stdout {
codec => rubydebug
}
1 # Sample Logstash configuration for creating a simple
2 # Beats -> Logstash -> Elasticsearch pipeline.
3
4 input {
5 beats {
6 port => 5044
7 }
8 }
9
10 output {
11 elasticsearch {
12 hosts => ["http://10.0.8.5:9200"]
13 index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
14 #user => "elastic"
15 #password => "changeme"
16 }
17 stdout {
18 codec => rubydebug
19 }
20 }
5. 启动logstash测试¶
# 会占用此窗口，将所有信息打印到窗口！
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-to-elasticsearch.conf
root@4c16g:/home/ubuntu# /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/logstash-to-elasticsearch.conf
Using bundled JDK: /usr/share/logstash/jdk
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
[INFO ] 2022-01-01 21:41:11.140 [main] runner - Starting Logstash {"logstash.version"=>"7.16.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[WARN ] 2022-01-01 21:41:11.554 [LogStash::Runner] multilocal - Ignoring the 'pipelines.yml' file because modules or command line options are specified
[INFO ] 2022-01-01 21:41:13.447 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
[INFO ] 2022-01-01 21:41:14.518 [Converge PipelineAction::Create<main>] Reflections - Reflections took 101 ms to scan 1 urls, producing 119 keys and 417 values
[WARN ] 2022-01-01 21:41:15.476 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-01-01 21:41:15.514 [Converge PipelineAction::Create<main>] beats - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-01-01 21:41:15.565 [Converge PipelineAction::Create<main>] plain - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[WARN ] 2022-01-01 21:41:15.671 [Converge PipelineAction::Create<main>] elasticsearch - Relying on default value of `pipeline.ecs_compatibility`, which may change in a future major release of Logstash. To avoid unexpected changes when upgrading Logstash, please explicitly declare your desired ECS Compatibility mode.
[INFO ] 2022-01-01 21:41:15.944 [[main]-pipeline-manager] elasticsearch - New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://10.0.8.5:9200"]}
[INFO ] 2022-01-01 21:41:16.314 [[main]-pipeline-manager] elasticsearch - Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://10.0.8.5:9200/]}}
[WARN ] 2022-01-01 21:41:16.598 [[main]-pipeline-manager] elasticsearch - Restored connection to ES instance {:url=>"http://10.0.8.5:9200/"}
[INFO ] 2022-01-01 21:41:16.614 [[main]-pipeline-manager] elasticsearch - Elasticsearch version determined (7.16.2) {:es_version=>7}
[WARN ] 2022-01-01 21:41:16.617 [[main]-pipeline-manager] elasticsearch - Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>7}
[INFO ] 2022-01-01 21:41:16.814 [Ruby-0-Thread-10: :1] elasticsearch - Using a default mapping template {:es_version=>7, :ecs_compatibility=>:disabled}
[INFO ] 2022-01-01 21:41:16.875 [[main]-pipeline-manager] javapipeline - Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50, "pipeline.max_inflight"=>500, "pipeline.sources"=>["/etc/logstash/conf.d/logstash-to-elasticsearch.conf"], :thread=>"#<Thread:0xde08d29 run>"}
[INFO ] 2022-01-01 21:41:16.880 [Ruby-0-Thread-10: :1] elasticsearch - Installing Elasticsearch template {:name=>"logstash"}
[INFO ] 2022-01-01 21:41:18.269 [[main]-pipeline-manager] javapipeline - Pipeline Java execution initialization time {"seconds"=>1.39}
[INFO ] 2022-01-01 21:41:18.325 [[main]-pipeline-manager] beats - Starting input listener {:address=>"0.0.0.0:5044"}
[INFO ] 2022-01-01 21:41:18.344 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-01-01 21:41:18.414 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2022-01-01 21:41:18.563 [[main]<beats] Server - Starting server on port: 5044
1. 如果提示内存不足！¶
# 修改配置文件，减少jdk内存数值
vim /etc/logstash/jvm.options
# 修改6-7行，从1g改为100m-500m测试！
1 ## JVM configuration
2
3 # Xms represents the initial size of total heap space
4 # Xmx represents the maximum size of total heap space
5
6 -Xms100m
7 -Xmx500m
2. 如果提示找不到logstash.yml¶
systemctl daemon-reload
6. 启动关闭命令¶
# deb包安装的，可以直接用systemctl启动关闭！
systemctl status logstash
systemctl start logstash
systemctl stop logstash
最后更新:
2022-02-19 13:05:46