容器服务(KCE)Nginx-ingress支持-金山云

本文将为您介绍基于金山云容器服务搭建 Nginx-ingress 服务的操作指南。

Nginx-ingress controller部署

使用Nginx-ingress服务的前提是在集群内部署nginx-ingress controller，controller部署的nginx-install.yaml如下：

apiVersion: v1
kind: Namespace
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: tcp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: udp-services
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: nginx-ingress-serviceaccount
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: nginx-ingress-clusterrole
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
      - "networking.k8s.io"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
  name: nginx-ingress-role
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
  name: nginx-ingress-role-nisa-binding
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: nginx-ingress-role
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: nginx-ingress-clusterrole-nisa-binding
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: nginx-ingress-clusterrole
subjects:
  - kind: ServiceAccount
    name: nginx-ingress-serviceaccount
    namespace: ingress-nginx

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-ingress-controller
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: ingress-nginx
      app.kubernetes.io/part-of: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ingress-nginx
        app.kubernetes.io/part-of: ingress-nginx
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
    spec:
      # wait up to five minutes for the drain of connections
      terminationGracePeriodSeconds: 300
      serviceAccountName: nginx-ingress-serviceaccount
      nodeSelector:
        kubernetes.io/os: linux
      containers:
        - name: nginx-ingress-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.30.0
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/udp-services
            - --publish-service=$(POD_NAMESPACE)/ingress-nginx
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 101
            runAsUser: 101
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
              protocol: TCP
            - name: https
              containerPort: 443
              protocol: TCP
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          lifecycle:
            preStop:
              exec:
                command:
                  - /wait-shutdown

---

apiVersion: v1
kind: LimitRange
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
spec:
  limits:
  - min:
      memory: 90Mi
      cpu: 100m
    type: Container

执行以下命令安装部署nginx-ingress controller 0.30.0版本：

[[email protected] ~]# kubectl apply -f nginx-install.yaml

#将会在集群中创建如下资源
namespace/ingress-nginx created
configmap/nginx-configuration created
configmap/tcp-services created
configmap/udp-services created
serviceaccount/nginx-ingress-serviceaccount created
clusterrole.rbac.authorization.k8s.io/nginx-ingress-clusterrole created
role.rbac.authorization.k8s.io/nginx-ingress-role created
rolebinding.rbac.authorization.k8s.io/nginx-ingress-role-nisa-binding created
clusterrolebinding.rbac.authorization.k8s.io/nginx-ingress-clusterrole-nisa-binding created
deployment.apps/nginx-ingress-controller created
limitrange/ingress-nginx created

为Deployment nginx-ingress-controller创建一个类型为LoadBalancer的Service，将通过金山云的LB暴露到集群外。nginx-ingress-svc.yaml如下：

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx
  name: nginx-ingress
spec:
  ports:
  - name: "tcp-80-80"
    port: 80
    protocol: TCP
    targetPort: 80
  - name: "tcp-443-443"
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app.kubernetes.io/name: "ingress-nginx"
    app.kubernetes.io/part-of: "ingress-nginx"
  type: LoadBalancer

创建服务，并获取服务的IP地址。

[[email protected] ~]# kubectl apply -f nginx-ingress-svc.yaml -n ingress-nginx
service/nginx-ingress created

[[email protected] ~]# kubectl get svc -n ingress-nginx
NAME            TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
nginx-ingress   LoadBalancer   10.254.137.163   120.131.xx.xx   80:30563/TCP,443:30681/TCP   17s

通过EXTERNAL-IP（120.131.xx.xx），外部流量将访问到集群中的nginx-ingress-controller，进而实现Ingress规则中的路由转发。

[[email protected] ~]# kubectl get all -n ingress-nginx
NAME                                            READY   STATUS    RESTARTS   AGE
pod/nginx-ingress-controller-7fcf8df75d-pqbgd   1/1     Running   0          132m

NAME                    TYPE           CLUSTER-IP       EXTERNAL-IP     PORT(S)                      AGE
service/nginx-ingress   LoadBalancer   10.254.137.163   120.131.xx.xx   80:30563/TCP,443:30681/TCP   2m52s

NAME                                       READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-ingress-controller   1/1     1            1           149m

NAME                                                  DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-ingress-controller-7fcf8df75d   1         1         1       149m

创建测试应用

以下创建两个应用，用于测试。

hello-world.yaml如下：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world
        image: hub.kce.ksyun.com/kingsoft/hello-world:latest
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: hello-world
  name: hello-world-svc
spec:
  ports:
  - port: 80
    protocol: TCP
    targetPort: 80
  selector:
    app: hello-world
  type: ClusterIP

hello-k8s.yaml如下：

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-k8s
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-k8s
  template:
    metadata:
      labels:
        app: hello-k8s
    spec:
      containers:
      - name: hello-k8s
        image: hub.kce.ksyun.com/kingsoft/hello-k8s:latest
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: hello-k8s
  name: hello-k8s-svc
spec:
  ports:
  - port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: hello-k8s
  type: ClusterIP

创建对应的deploy和service：

[[email protected] hello]# kubectl create -f hello-k8s.yaml 
deployment.extensions/hello-k8s created
service/hello-k8s-svc created

[[email protected] hello]# kubectl create -f hello-world.yaml 
deployment.extensions/hello-world created
service/hello-world-svc created

[[email protected] hello]# kubectl get deploy
NAME          DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
hello-k8s     1         1         1            1           5m2s
hello-world   1         1         1            1           4m50s

[[email protected] hello]# kubectl get svc
NAME              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
hello-k8s-svc     ClusterIP   10.254.131.29   <none>        8080/TCP   5m31s
hello-world-svc   ClusterIP   10.254.244.96   <none>        80/TCP     5m19s
kubernetes        ClusterIP   10.254.0.1      <none>        443/TCP    52d

Ingress配置策略

为了支持灵活的分发策略，ingress策略可以按照多种分发方式进行配置，下面对几种常见的ingress转发策略简单介绍。

同一域名下，不同的URL的路径转发到不同服务上

这种配置常用于一个网站通过不同的路径提供不同服务的场景。

通过如下的访问配置：

对 http://my.nginx.test/hello-k8s 的访问将被路由到后端名为"hello-k8s-svc" 的Service。
对 http://my.nginx.test/hello-world 的访问将被路由到后端名为"hello-world-svc" 的Service。

ingress.yaml如下：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-test
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: my.nginx.test
    http:
      paths:
      - path: /hello-world
        backend:
          serviceName: hello-world-svc
          servicePort: 80
      - path: /hello-k8s
        backend:
          serviceName: hello-k8s-svc
          servicePort: 8080

创建ingress规则：

[[email protected] ~]# kubectl apply -f ingress.yaml
ingress.extensions/nginx-test created

[[email protected] ~]# kubectl get ingress
NAME         HOSTS           ADDRESS   PORTS   AGE
nginx-test   my.nginx.test             80      15s

备注：

这里我们将自有域名my.nginx.test解析到负载均衡的IP。
Ingress规则与nginx-ingress-controller的对应关系通过注解kubernetes.io/ingress.class: nginx来指定，表示此条ingress规则由nginx-ingress-controller处理。

在浏览器的访问验证如下：

Nginx-ingress支持

不同的域名转发到不同的服务

这种配置常用于一个网站通过不同的域名或者虚拟主机名提供不同的服务的场景。

通过如下的访问配置：

对 http://nginx.hello.k8s 的访问将被路由到后端名为"hello-k8s-svc" 的Service。
对 http://nginx.hello.world 的访问将被路由到后端名为"hello-world-svc" 的Service。

ingress2.yaml如下：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-test-2
  annotations:
    kubernetes.io/ingress.class: nginx
spec:
  rules:
  - host: nginx.hello.k8s
    http:
      paths:
      - path: /  
        backend:
          serviceName: hello-k8s-svc
          servicePort: 8080
  - host: nginx.hello.world
    http:
      paths:
      - path: /  
        backend:
          serviceName: hello-world-svc
          servicePort: 80

创建Ingress规则：

[[email protected] ~]# kubectl apply -f ingress2.yaml
ingress.extensions/nginx-test-2 created

[[email protected] ~]# kubectl get ingress
NAME           HOSTS                               ADDRESS   PORTS   AGE
nginx-test-2   nginx.hello.k8s,nginx.hello.world             80      19s

在浏览器的访问验证如下：

Nginx-ingress支持

HTTPS访问

当Ingress配置TLS时，服务将以HTTPS协议的方式对外暴露。

准备证书

这里我们作为测试用例使用自签名证书，使用如下命令快速创建。

[[email protected] ~]# openssl req -newkey rsa:2048 -nodes -keyout tls.key -x509 -days 365 -out tls.crt
Generating a 2048 bit RSA private key
...............+++
.............................+++
writing new private key to 'tls.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:CN
State or Province Name (full name) []:Beijing
Locality Name (eg, city) [Default City]:Beijing
Organization Name (eg, company) [Default Company Ltd]:Kingsoft
Organizational Unit Name (eg, section) []:Ksyun
Common Name (eg, your name or your server's hostname) []:nginx.hello.k8s    #配置需使用证书的域名
Email Address []:[email protected]

将会在当前路径下创建证书(tls.crt)和私钥文件(tls.key)。

创建Secret资源

根据现有证书和私钥创建Secret资源，将会创建一个类型为kubernetes.io/tls的Secret。

[[email protected] ~]# kubectl create secret tls secret-https --key tls.key --cert tls.crt
secret/secret-https created

创建Ingress规则

对上述示例中的http访问方式进行升级，实现对nginx.hello.k8s的https访问。ingress-https.yaml如下：

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: nginx-test-2
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"    #开启重定向功能
spec:
  rules:
  - host: nginx.hello.k8s
    http:
      paths:
      - path: /  
        backend:
          serviceName: hello-k8s-svc
          servicePort: 8080
  tls:
  - hosts:
    - nginx.hello.k8s
    secretName: secret-https

在浏览器的访问验证如下：

Nginx-ingress支持

更多Nginx ingress controller的特性，请参考NGINX Ingress Controller。

文档中心

产品简介

产品概述

购买与计费

快速入门

用户指南

授权管理

集群管理

多集群

节点管理

Helm应用

工作负载管理

自动伸缩

服务管理

配置管理

存储卷

镜像仓库

日志

监控与告警

Serverless集群用户指南

集群管理

工作负载管理

服务管理

存储卷

KCI Pod的管理

容器服务实践

迁移

发布

API文档

调用方式

集群相关接口

镜像仓库相关接口