全部文档
当前文档
最近更新时间：2024-04-12 14:02:16
对于通过filebeat采集容器实例日志至Kafka服务的场景，若您对filebeat有自定义需求，可通过如下方式进行配置。
已在Kubernetes集群中部署虚拟节点，部署方式：KCE集群参考Kubernetes集群对接KCI，自建集群参考自建Kubernetes集群中对接KCI。
容器实例所属VPC已与Kafka集群所属网络打通。
注：若Kafka集群有安全组配置，入站规则中需配置放行broker监听端口。
目标采集的容器实例日志类型为容器文件日志，自定义sidecar方式下不支持采集容器标准输出日志。
在集群Kube-system命名空间下创建configmap filebeat-config用于配置Kafka output。
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-config
namespace: kube-system
data:
filebeat.yml: |
---
filebeat.config:
inputs:
path: "${path.config}/inputs.d/*.yml"
reload.enabled: true
reload.period: "10s"
modules:
path: "${path.config}/modules.d/*.yml"
reload.enabled: true
output.kafka:
# 配置Kafka broker地址
hosts: ["10.0.0.***:9092", "10.0.0.***:9092", "10.0.0.***:9092"]
# 动态匹配topic地址 + 分区配置
topic: '%{[fields.log_topic]}'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000注：更多Kafka output配置可参考filebeat官网文档Configure the Kafka output。
在集群kube-system命名空间下创建configmap用于自定义sidecar配置，yaml示例如下：
注：
configmap名称与对应虚拟节点同名。
配置key必须为config.yaml。
apiVersion: v1
kind: ConfigMap
metadata:
name: rbkci-virtual-kubelet
namespace: kube-system
data:
config.yaml: |
mutation:
customMutation:
containers: #自定义sidecar容器配置
- args:
- -c
- /usr/share/filebeat/config/filebeat.yml
- -e
command:
- /usr/share/filebeat/filebeat
image: docker.elastic.co/beats/filebeat:7.17.0 #自定义filebeat镜像
imagePullPolicy: Always
name: filebeat
volumeMounts:
- mountPath: /usr/share/filebeat/config
name: filebeat-config
- mountPath: /usr/share/filebeat/inputs.d
name: filebeat-inputs
- mountPath: /usr/share/filebeat/data
name: filebeat-data
- mountPath: /home/q/logs/collected #自定义业务容器日志的hostPath路径
name: filebeat-logdir
SecurityContext:
runAsUser: 0
volumes: #对应container的volume定义
- configMap:
name: filebeat-config
name: filebeat-config
- configMap:
name: filebeat-inputs
name: filebeat-inputs
- hostPath:
path: /usr/share/filebeat/data
type: DirectoryOrCreate
name: filebeat-data
- hostPath:
path: /home/q/logs/collected
name: filebeat-logdir容器实例通过CoreDNS服务解析消费端地址，Kafka服务端域名需通过集群Coredns hosts配置，示例如下：
apiVersion: v1
data:
Corefile: |
.:53 {
errors
health
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
fallthrough in-addr.arpa ip6.arpa
}
# hosts can add hosts's item into dns, see https://coredns.io/plugins/hosts/
hosts {
198.18.96.191 hub.kce.ksyun.com
10.0.0.*** kmr-c0b4eaab-gn-e2a4babf-broker-1-1.ksc.com // kafka broker 域名
10.0.0.*** kmr-c0b4eaab-gn-e2a4babf-broker-1-2.ksc.com // kafka broker 域名
10.0.0.*** kmr-c0b4eaab-gn-e2a4babf-broker-1-3.ksc.com // kafka broker 域名
fallthrough
}
prometheus :9153
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
kind: ConfigMap
metadata:
creationTimestamp: "2021-12-15T11:14:52Z"
name: coredns
namespace: kube-system
resourceVersion: "6152795"
uid: c1e29f37-d37d-4c90-9ca4-418a628cc04b在集群kube-system命名空间下创建configmap filebeat-inputs:
apiVersion: v1
kind: ConfigMap
metadata:
name: filebeat-inputs
namespace: kube-system
data:
kci.yml: |
---
- type: "log"
symlinks: true
enabled: true
fields:
log_topic: filelog
paths:
- "/home/q/logs/collected/*.log" #指定日志采集文件路径以下以nginx pod为例，通过定义annotation，为pod开启kube-proxy。
注：
开启Kube-proxy：容器实例需通过CoreDNS服务解析消费端地址，需开启Kube-proxy以使能pod访问ClusterIP类型服务。
关闭Klog配置：暂不支持同时开启klog日志采集与自定义sidecar日志采集。
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-rbkci
namespace: default
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
annotations:
k8s.ksyun.com/kci-kube-proxy-enabled: "true" #开启Kube-proxy
labels:
app: nginx
spec:
containers:
- name: nginx
image: nginx:latest
volumeMounts:
- name: rbkcilog
mountPath: /var/log/nginx #将底层目录"/home/q/logs/collected"挂载到容器"/var/log/nginx"路径下
volumes:
- hostPath:
path: /home/q/logs/collected #对应自定义filebeat的日志采集路径
name: rbkcilog
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: type
operator: In
values:
- virtual-kubelet
tolerations:
- key: rbkci-virtual-kubelet.io/provider
value: kingsoftcloud
effect: NoSchedule若需要在虚拟节点维度开启自定义日志采集，可修改virtual-kubelet启动参数，在vk级别指定自定义镜像及开启kube-proxy，示例如下：
apiVersion: apps/v1
kind: Deployment
metadata:
name: rbkci-virtual-kubelet
namespace: kube-system
labels:
k8s-app: rbkci-virtual-kubelet
spec:
replicas: 1
selector:
matchLabels:
k8s-app: rbkci-virtual-kubelet
template:
metadata:
name: rbkci-virtual-kubelet
labels:
k8s-app: rbkci-virtual-kubelet
spec:
serviceAccountName: virtual-kubelet-sa
containers:
- name: virtual-kubelet
image: hub.kce.ksyun.com/ksyun/rbkci-virtual-kubelet:v1.1.0-beta
args:
- --nodename=rbkci-virtual-kubelet
- --cluster-dns=10.254.0.10
- --cluster-domain=cluster.local
- --kcilet-kubeconfig-path=/root/.kube/config
- --enable-node-lease
# 虚拟节点管理的所有实例使能kube-proxy
- --kube-proxy-enable
imagePullPolicy: Always
env:
- name: VKUBELET_POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
- name: TEMP_AKSK_CM
value: user-temp-aksk
- name: KCI_CLUSTER_ID
value: ${cluster_id}
- name: KCI_SUBNET_ID
value: ${subnet_id}
- name: KCI_SECURITY_GROUP_IDS
value: ${security_group_ids}
# 指定虚拟节点管理的所有实例底层镜像
- name: KCI_BASE_IMAGE
value: ${kci_base_image}
volumeMounts:
- mountPath: /root/.kube
name: kubeconfig
- mountPath: /var/log/kci-virtual-kubelet
name: kci-provider-log
volumes:
- name: kubeconfig
secret:
secretName: rbkci-kubeconfig-secret
- name: kci-provider-log
hostPath:
path: /var/log/kci-virtual-kubelet模拟容器日志，查询Kafka消费端消息，检查目标容器实例日志是否投递成功。
从上图中可以看到，filebeat的版本（7.17.0）和采集路径（/home/q/logs/collected）已与自定义sidecar的配置相同。
纯净模式