kubernetes设置

  1. k8s 1.10之前需要在kube-apiserver、kube-controller-manager、kube-scheduler、kubelet中开启如下feature,如果不是首次部署的话,重启以上所有组件:

    –feature-gates=“DevicePlugins=true”

  2. 安装 NVIDIA Driver~=361.93

  3. 安装nvidia-docker2,由于目前使用的docker版本提示信息里包含自定义字样,只能使用如下方式安装:

    所有安装方式参考这里

     1
     2
     3
     4
     5
     6
     7
     8
     9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    
    # If you have nvidia-docker 1.0 installed: we need to remove it and all existing GPU containers
      docker volume ls -q -f driver=nvidia-docker | xargs -r -I{} -n1 docker ps -q -a -f volume={} | xargs -r docker rm -f
      sudo yum remove nvidia-docker
          
      # Add the package repositories
      distribution=$(. /etc/os-release;echo $ID$VERSION_ID)
      curl -s -L https://nvidia.github.io/nvidia-container-runtime/$distribution/nvidia-container-runtime.repo | \
        sudo tee /etc/yum.repos.d/nvidia-container-runtime.repo
          
      # Install the nvidia runtime hook
      sudo yum install -y nvidia-container-runtime-hook
      sudo mkdir -p /usr/libexec/oci/hooks.d
      echo -e '#!/bin/sh\nPATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" exec nvidia-container-runtime-hook "$@"' | \
        sudo tee /usr/libexec/oci/hooks.d/nvidia
      sudo chmod +x /usr/libexec/oci/hooks.d/nvidia
          
      # Test nvidia-smi with the latest official CUDA image
      # You can't use `--runtime=nvidia` with this setup.
      docker run --rm nvidia/cuda nvidia-smi
  4. 安装nvidia-container-runtime,在上一步中已经安装了对应的yum repo,这里直接执行如下命令即可:

    因为使用了上一步的安装方式,所以需要进行这一步的安装,如果是通过yum直接安装的nvidia-docker2,则不需要进行此步。

    1
    2
    
    # install runtime
    yum install nvidia-container-runtime
  5. update docker daemon,在docker daemon中添加如下配置

    daemon.json中添加如下配置,可选配置为”default-runtime”: “nvidia”,如果不设置默认runtime,则默认使用runc,启动容器是需要指定–runtime=nvidia

    1
    2
    3
    4
    5
    6
    7
    
    "default-runtime": "nvidia""runtimes": {
           "nvidia": {
               "path": "/usr/bin/nvidia-container-runtime",
               "runtimeArgs": []
           }
       }
  6. 安装NVIDIA device plugin,插件以daemonset方式部署,如果集群中既有CPU也有GPU节点,可以通过label筛选出GPU节点,无GPU的节点无需部署此程序

    1
    2
    3
    4
    
    # For Kubernetes v1.8
    kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.8/nvidia-device-plugin.yml
    # For Kubernetes v1.9
    kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v1.9/nvidia-device-plugin.yml

测试

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
   apiVersion: v1
   kind: Pod
   metadata:
     name: cuda-vector-add
   spec:
     restartPolicy: OnFailure
     containers:
       - name: cuda-vector-add
         # https://github.com/kubernetes/kubernetes/blob/v1.7.11/test/images/nvidia-cuda/Dockerfile
         image: "k8s.gcr.io/cuda-vector-add:v0.1"
         resources:
           limits:
             nvidia.com/gpu: 1
     nodeSelector:
       GPU: "true" //测试时自己给对应GPU节点加了GPU=true的label
       
   kubectl create -f test.yaml

执行结果:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
   [root@vm10-0-13-17 ~]# kubectl get pods -a
   NAME                                   READY     STATUS      RESTARTS   AGE
   cuda-vector-add                        0/1       Completed   0          3h
   nvidia-device-plugin-daemonset-pv6z8   1/1       Running     0          4h
   [root@vm10-0-13-17 ~]# kubectl logs cuda-vector-add
   [Vector addition of 50000 elements]
   Copy input data from the host memory to the CUDA device
   CUDA kernel launch with 196 blocks of 256 threads
   Copy output data from the CUDA device to the host memory
   Test PASSED
   Done