全部文档
当前文档

暂无内容

如果没有找到您期望的内容,请尝试其他搜索词

文档中心

自定义sidecar方式采集容器实例日志

最近更新时间:2023-10-09 20:08:01

对于通过filebeat采集容器实例日志至Kafka服务的场景,若您对filebeat有自定义需求,可通过如下方式进行配置。

前提条件

  1. 已在Kubernetes集群中部署虚拟节点,部署方式:KCE集群参考Kubernetes集群对接KCI,自建集群参考自建Kubernetes集群中对接KCI

  2. 容器实例所属VPC已与Kafka集群所属网络打通。

    注:若Kafka集群有安全组配置,入站规则中需配置放行broker监听端口。

  3. 目标采集的容器实例日志类型为容器文件日志,自定义sidecar方式下不支持采集容器标准输出日志。

步骤1:创建filebeat配置文件

在集群Kube-system命名空间下创建configmap filebeat-config用于配置Kafka output。

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
data:
  filebeat.yml: |
    ---
    filebeat.config:
      inputs:
        path: "${path.config}/inputs.d/*.yml"
        reload.enabled: true
        reload.period: "10s"
      modules:
        path: "${path.config}/modules.d/*.yml"
        reload.enabled: true
    output.kafka: 
      # 配置Kafka broker地址
      hosts: ["10.0.0.***:9092", "10.0.0.***:9092", "10.0.0.***:9092"]
      
      # 动态匹配topic地址 + 分区配置
      topic: '%{[fields.log_topic]}'
      partition.round_robin:
      reachable_only: false
 
      required_acks: 1
      compression: gzip
      max_message_bytes: 1000000

注:更多Kafka output配置可参考filebeat官网文档Configure the Kafka output

步骤2:创建自定义sidecar配置

在集群kube-system命名空间下创建configmap用于自定义sidecar配置,yaml示例如下:

注:

  1. configmap名称与对应虚拟节点同名。
  2. 配置key必须为config.yaml。
apiVersion: v1
kind: ConfigMap
metadata:
  name: rbkci-virtual-kubelet
  namespace: kube-system
data:
  config.yaml: |
    mutation:
      customMutation:
        containers:    #自定义sidecar容器配置
          - args:
              - -c
              - /usr/share/filebeat/config/filebeat.yml
              - -e
            command:
              - /usr/share/filebeat/filebeat
            image: docker.elastic.co/beats/filebeat:7.17.0    #自定义filebeat镜像
            imagePullPolicy: Always
            name: filebeat
            volumeMounts:
              - mountPath: /usr/share/filebeat/config
                name: filebeat-config
              - mountPath: /usr/share/filebeat/inputs.d
                name: filebeat-inputs
              - mountPath: /usr/share/filebeat/data
                name: filebeat-data
              - mountPath: /home/q/logs/collected   #自定义业务容器日志的hostPath路径
                name: filebeat-logdir
            SecurityContext:
              runAsUser: 0
        volumes:    #对应container的volume定义
          - configMap:
              name: filebeat-config
            name: filebeat-config
          - configMap:
              name: filebeat-inputs
            name: filebeat-inputs
          - hostPath:
              path: /usr/share/filebeat/data
              type: DirectoryOrCreate
            name: filebeat-data
          - hostPath:
              path: /home/q/logs/collected 
            name: filebeat-logdir

步骤3:配置Kafka服务端域名解析

容器实例通过CoreDNS服务解析消费端地址,Kafka服务端域名需通过集群Coredns hosts配置,示例如下:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        # hosts can add hosts's item into dns, see https://coredns.io/plugins/hosts/
        hosts {
            198.18.96.191 hub.kce.ksyun.com
            10.0.0.*** kmr-c0b4eaab-gn-e2a4babf-broker-1-1.ksc.com  // kafka broker 域名
            10.0.0.*** kmr-c0b4eaab-gn-e2a4babf-broker-1-2.ksc.com  // kafka broker 域名
            10.0.0.*** kmr-c0b4eaab-gn-e2a4babf-broker-1-3.ksc.com  // kafka broker 域名
            fallthrough
        }
        prometheus :9153
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  creationTimestamp: "2021-12-15T11:14:52Z"
  name: coredns
  namespace: kube-system
  resourceVersion: "6152795"
  uid: c1e29f37-d37d-4c90-9ca4-418a628cc04b

步骤4: 配置日志采集规则

在集群kube-system命名空间下创建configmap filebeat-inputs:

apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-inputs
  namespace: kube-system
data:
  kci.yml: |
    ---
    - type: "log"   
      symlinks: true
      enabled: true
      fields:
        log_topic: filelog
      paths:
      - "/home/q/logs/collected/*.log"    #指定日志采集文件路径

示例:为容器实例开启自定义sidecar日志采集

以下以nginx pod为例,通过定义annotation,为pod开启kube-proxy。

注:

  1. 开启Kube-proxy:容器实例需通过CoreDNS服务解析消费端地址,需开启Kube-proxy以使能pod访问ClusterIP类型服务。
  2. 关闭Klog配置:暂不支持同时开启klog日志采集与自定义sidecar日志采集。
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-rbkci
  namespace: default
  labels:
    app: nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
        annotations:
            k8s.ksyun.com/kci-kube-proxy-enabled: "true"     #开启Kube-proxy
        labels:
            app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx:latest
        volumeMounts:
            - name: rbkcilog
              mountPath: /var/log/nginx   #将底层目录"/home/q/logs/collected"挂载到容器"/var/log/nginx"路径下
      volumes:
        - hostPath:
            path: /home/q/logs/collected  #对应自定义filebeat的日志采集路径
          name: rbkcilog
      nodeName: rbkci-virtual-kubelet 

若需要在虚拟节点维度开启自定义日志采集,可修改virtual-kubelet启动参数,在vk级别指定自定义镜像及开启kube-proxy,示例如下:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rbkci-virtual-kubelet
  namespace: kube-system
  labels:
    k8s-app: rbkci-virtual-kubelet
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: rbkci-virtual-kubelet
  template:
    metadata:
      name: rbkci-virtual-kubelet
      labels:
        k8s-app: rbkci-virtual-kubelet
    spec:
      serviceAccountName: virtual-kubelet-sa
      containers:
        - name: virtual-kubelet
          image: hub.kce.ksyun.com/ksyun/rbkci-virtual-kubelet:v1.1.0-beta
          args:
            - --nodename=rbkci-virtual-kubelet
            - --cluster-dns=10.254.0.10
            - --cluster-domain=cluster.local
            - --kcilet-kubeconfig-path=/root/.kube/config
            - --enable-node-lease
            # 虚拟节点管理的所有实例使能kube-proxy
            - --kube-proxy-enable
          imagePullPolicy: Always
          env:
            - name: VKUBELET_POD_IP
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
            - name: TEMP_AKSK_CM
              value: user-temp-aksk
            - name: KCI_CLUSTER_ID
              value: ${cluster_id}
            - name: KCI_SUBNET_ID
              value: ${subnet_id}
            - name: KCI_SECURITY_GROUP_IDS
              value: ${security_group_ids}
            # 指定虚拟节点管理的所有实例底层镜像
            - name: KCI_BASE_IMAGE
              value: ${kci_base_image}
          volumeMounts:
            - mountPath: /root/.kube
              name: kubeconfig
            - mountPath: /var/log/kci-virtual-kubelet
              name: kci-provider-log
      volumes:
        - name: kubeconfig
          secret:
            secretName: rbkci-kubeconfig-secret
        - name: kci-provider-log
          hostPath:
            path: /var/log/kci-virtual-kubelet

验证日志投递效果

模拟容器日志,查询Kafka消费端消息,检查目标容器实例日志是否投递成功。
image20220523114550882.png

从上图中可以看到,filebeat的版本(7.17.0)和采集路径(/home/q/logs/collected)已与自定义sidecar的配置相同。

文档导读
纯净模式常规模式

纯净模式

点击可全屏预览文档内容
文档反馈