全部文档
当前文档

暂无内容

如果没有找到您期望的内容,请尝试其他搜索词

文档中心

使用NodeLocal DNSCache

最近更新时间:2024-09-18 15:57:15

NodeLocal DNSCache 通过在集群节点上作为 DaemonSet 运行 DNS 缓存代理来提高集群 DNS 性能。当集群规模较大时,建议通过使用NodeLocal DNSCache来提升服务发现的稳定性和性能。

功能说明

  • 使用当前的 DNS 体系结构,如果没有本地 kube-dns/CoreDNS 实例,则具有最高 DNS QPS 的 Pod 可能必须延伸到另一个节点。 在这种场景下,拥有本地缓存将有助于改善延迟。

  • 跳过 iptables DNAT 和连接跟踪将有助于减少conntrack竞争并避免 UDP DNS 条目填满 conntrack 表。

  • 从本地缓存代理到 kube-dns 服务的连接可以升级为 TCP。 TCP conntrack 条目将在连接关闭时被删除,相反 UDP 条目必须超时 (默认nf_conntrack_udp_timeout 是 30 秒)。

  • 将 DNS 查询从 UDP 升级到 TCP 将减少由于被丢弃的 UDP 包和 DNS 超时而带来的尾部等待时间; 这类延时通常长达 30 秒(3 次重试 + 10 秒超时)。 由于 nodelocal 缓存监听 UDP DNS 查询,应用不需要变更。

  • 在节点级别对 DNS 请求的度量和可见性。

  • 可以重新启用负缓存,从而减少对 kube-dns 服务的查询数量。

部署模版

node-local-dns 部署模版:

# Copyright 2018 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

apiVersion: v1
kind: ServiceAccount
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
---
apiVersion: v1
kind: Service
metadata:
  name: kube-dns-upstream
  namespace: kube-system
  labels:
    k8s-app: kube-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
    kubernetes.io/name: "KubeDNSUpstream"
spec:
  ports:
  - name: dns
    port: 53
    protocol: UDP
    targetPort: 53
  - name: dns-tcp
    port: 53
    protocol: TCP
    targetPort: 53
  selector:
    k8s-app: coredns
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  Corefile: |
    __PILLAR__DNS__DOMAIN__:53 {
        errors
        cache {
                success 9984 30
                denial 9984 5
        }
        reload
        loop
        bind __PILLAR__LOCAL__DNS__ __PILLAR__DNS__SERVER__
        forward . __PILLAR__CLUSTER__DNS__ {
                force_tcp
        }
        prometheus :9253
        health __PILLAR__LOCAL__DNS__:8081
        }
    in-addr.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind __PILLAR__LOCAL__DNS__ __PILLAR__DNS__SERVER__
        forward . __PILLAR__CLUSTER__DNS__ {
                force_tcp
        }
        prometheus :9253
        }
    ip6.arpa:53 {
        errors
        cache 30
        reload
        loop
        bind __PILLAR__LOCAL__DNS__ __PILLAR__DNS__SERVER__
        forward . __PILLAR__CLUSTER__DNS__ {
                force_tcp
        }
        prometheus :9253
        }
    .:53 {
        errors
        cache 30
        reload
        loop
        bind __PILLAR__LOCAL__DNS__ __PILLAR__DNS__SERVER__
        forward . __PILLAR__UPSTREAM__SERVERS__
        prometheus :9253
        }
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-local-dns
  namespace: kube-system
  labels:
    k8s-app: node-local-dns
    kubernetes.io/cluster-service: "true"
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  updateStrategy:
    rollingUpdate:
      maxUnavailable: 10%
  selector:
    matchLabels:
      k8s-app: node-local-dns
  template:
    metadata:
      labels:
        k8s-app: node-local-dns
      annotations:
        prometheus.io/port: "9253"
        prometheus.io/scrape: "true"
    spec:
      priorityClassName: system-node-critical
      serviceAccountName: node-local-dns
      hostNetwork: true
      dnsPolicy: Default  # Don't use cluster DNS.
      tolerations:
      - key: "CriticalAddonsOnly"
        operator: "Exists"
      - effect: "NoExecute"
        operator: "Exists"
      - effect: "NoSchedule"
        operator: "Exists"
      containers:
      - name: node-cache
        image: hub.kce.ksyun.com/ksyun/k8s-dns-node-cache:1.22.8
        resources:
          requests:
            cpu: 25m
            memory: 5Mi
        args: [ "-localip", "__PILLAR__LOCAL__DNS__,__PILLAR__DNS__SERVER__", "-conf", "/etc/Corefile", "-upstreamsvc", "kube-dns-upstream" ]
        securityContext:
          privileged: true
        ports:
        - containerPort: 53
          name: dns
          protocol: UDP
        - containerPort: 53
          name: dns-tcp
          protocol: TCP
        - containerPort: 9253
          name: metrics
          protocol: TCP
        livenessProbe:
          httpGet:
            host: __PILLAR__LOCAL__DNS__
            path: /health
            port: 8081
          initialDelaySeconds: 60
          timeoutSeconds: 5
        volumeMounts:
        - mountPath: /run/xtables.lock
          name: xtables-lock
          readOnly: false
        - name: config-volume
          mountPath: /etc/coredns
        - name: kube-dns-config
          mountPath: /etc/kube-dns
      volumes:
      - name: xtables-lock
        hostPath:
          path: /run/xtables.lock
          type: FileOrCreate
      - name: kube-dns-config
        configMap:
          name: coredns
          optional: true
      - name: config-volume
        configMap:
          name: node-local-dns
          items:
            - key: Corefile
              path: Corefile.base
---
# A headless service is a service with a service IP but instead of load-balancing it will return the IPs of our associated Pods.
# We use this to expose metrics to Prometheus.
apiVersion: v1
kind: Service
metadata:
  annotations:
    prometheus.io/port: "9253"
    prometheus.io/scrape: "true"
  labels:
    k8s-app: node-local-dns
  name: node-local-dns
  namespace: kube-system
spec:
  clusterIP: None
  ports:
    - name: metrics
      port: 9253
      targetPort: 9253
  selector:
    k8s-app: node-local-dns

开始安装部署

  1. 将上文对应k8s版本的 node-local-dns 安装模板保存为 nodelocaldns.yaml 文件

  2. 按照如下方式生成对应的环境变量:

$ kubedns=`kubectl get svc coredns -n kube-system -o jsonpath={.spec.clusterIP}`
# 表示集群域,默认为 cluster.local,如果进行过修改,需要按修改值进行填写
$ domain=cluster.local
# 表示 DNSCache 本地的 IP,默认为169.254.20.10
$ localdns=169.254.20.10
  1. 按照以下命令替换模版文件中对应的变量

IPVS模式下(v1.15及以上集群)

sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/,__PILLAR__DNS__SERVER__//g; s/__PILLAR__CLUSTER__DNS__/$kubedns/g" nodelocaldns.yaml

IPTABLES 模式下(v1.15以下集群)

sed -i "s/__PILLAR__LOCAL__DNS__/$localdns/g; s/__PILLAR__DNS__DOMAIN__/$domain/g; s/__PILLAR__DNS__SERVER__/$kubedns/g" nodelocaldns.yaml
  1. 安装部署

kubectl apply -f nodelocaldns.yaml

使用node-local-dns

使用node-local-dns的解析有两种配置方式:

1. 配置Pod的dnsconfig。

2. 配置kubelet的cluster-dns。

配置Pod的dnsConfig

nameservers 指定 169.254.20.10

配置searches,使得集群内域名可以解析

ndots默认设置为5,可以适当降低以加速查询

#使用此方法,针对某个pod设置,不影响其他pod和组件
apiVersion: v1
kind: Pod
metadata:
  name: dnsutils
spec:
  dnsPolicy: None
  dnsConfig:
    nameservers:
      - 169.254.20.10
    searches:
      - default.svc.cluster.local # 注意:default为该pod所在的命名空间
      - svc.cluster.local
      - cluster.local
    options:
      - name: ndots
        value: "5"
  containers:
    - name: dnsutils
      image: hub.kce.ksyun.com/ksyun/jessie-dnsutils:1.3
      command:
        - sleep
        - "7200"
      imagePullPolicy: IfNotPresent
  restartPolicy: Always

配置kubelet的cluster-dns

使用此方法,生效的对象则是kubelet所管理的所有Pod,需要重启所有节点的kubelet。

  1. 修改kubelet配置,默认情况下配置为--cluster-dns=10.254.0.10(如果coredns service的IP有修改,按照修改为准)。将DNSCache本地的IP添加到coredns serviceIP之前,例如:--cluster-dns=10.254.0.10改成 --cluster-dns=169.254.20.10,10.254.0.10

  2. 执行systemctl restart kubelet ,重启kubelet

  3. 执行systemctl status kubelet ,确定kubelet重启成功。需要确保kubelet启动成功,如果kubelet长时间未启动会导致节点NotReady

注意:新增节点同样需要进行修改,如果新增节点需要统一配置,请提交工单

验证配置生效

部署以下busybox的实例deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: busybox
spec:
  selector:
    matchLabels:
      app: busybox
  replicas: 1
  template:
    metadata:
      labels:
        app: busybox
    spec:
      containers:
        - name: busybox
          image: hub.kce.ksyun.com/ksyun/busybox:latest
          args: ["sleep","100000"]

查看对应的pod运行状态:

kubectl get pod  -l app=busybox

NAME                       READY   STATUS    RESTARTS   AGE
busybox-64bdb5d64c-c****   1/1     Running   0          2m

查看对应pod的resolv.conf文件:

kubectl exec -it busybox-64bdb5d64c-c****  -- cat /etc/resolv.conf

search default.svc.cluster.local svc.cluster.local cluster.local
nameserver 169.254.20.10
nameserver 10.254.0.10
options ndots:5

nameserver中出现 169.254.20.10时,说明NodeLocal DNSCache 组件部署成功。

文档导读
纯净模式常规模式

纯净模式

点击可全屏预览文档内容
文档反馈