I am using the latest version of ELK Stack and I have Filebeat installed on different servers. I am using different filebeat modules to send the logs. Considering that each module handles a path configuration to the log files. How do I make according to what type of log or log source I want to add or custom fields or tags as metadata to then filter with kibana?

In the case of accessing the application server in glassfish, it created an input that includes the configuration: path, fields, tags from /etc/filebeat/filebeat.yml and it works. I understand that when enabling the modules it is not necessary to include the path of the logs in the inputs of filebeat.yml

But if I am using a different module (system, mysql, postgres, apache, nginx, etc.) to send records to logstash using filebeat: how do I insert custom fields or tags in the same way I would in filebeat.yml when I configure? The entries in the path of the records? Since each module handles its own configuration by default where it indicates that it even indicates the path of the log files. For this, I need to somehow conditionally detect the registry (apache, system, mysql, access.log, error.log, ip / hostname, application) that I am accessing to insert custom fields that I can use to filter later in kibana.

3 Answers

Had same problem, resolved it by using modules ingest pipeline with log input.

filebeat.inputs:
- type: log paths: - "/var/log/nginx/access.log*" fields: level: "INFO" application: "nginx" fields_under_root: true pipeline: "filebeat-7.3.0-nginx-access-default"

You need to load the ingest pipeline from module of course

Also you can modify nginx module setting as well.

I had same needs, so I modified nginx module conf.

it's under this folder (example for 7.3.1 filebeat)

access log

./filebeat/module/nginx/access/config/nginx-access.yml

error log

./filebeat/module/nginx/error/config/nginx-access.yml

add tags within the yml configuration file

type: log
paths:
{{ range $i, $path := .paths }} - {{$path}}
{{ end }}
exclude_files: [".gz$"]
#### TAGS HERE!!!!
tags: ["nginx", "anythinguwant"]
processors:
- add_locale: ~

For the record, the following is working for me (elk stack version 7.10.2, file /usr/share/filebeat/modules.d/nginx.yml):

- module: nginx access: enabled: true var.paths: ["/var/log/nginx/access.log*"] input: fields: custom.event.application: "nginx" custom.event.channel: "access"

Your Answer

Sign up or log in

Sign up using Google Sign up using Facebook Sign up using Email and Password Submit

Post as a guest

Post Your Answer Discard

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy