python记录NVIDIA显卡运行状态

前沿：由于此台服务器显卡经常全部丢失，所以需要持续记录显卡的运行状态。
1、系统：ubuntu18.04；
2、环境：python3.6；
3、查看显卡状态命令：“nvidia-smi”
4、后台运行 python 脚本：“nohup python3 -u nvidia.py > nvidia.out 2>&1 &”
5、开机自动运行脚本。
6、1.1-2021.01.04 版本日志：
①日志记录每天单独生成记录日志；
②判断显卡是否正常运行，若结果为空，循环等待；
③若检测到显卡丢失，则自动重启服务器。

1.0-2020.12.30 版本源代码如下：

#!/usr/bin/python3
import os
import datetime
import time

filepath = 'nvidia.log'
def nvidia_go():
    while True:
        f = open(filepath,'a')
        time_val = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
        f.write(time_val+"\n")
        val = os.popen("nvidia-smi")
        for v in val:
            f.write(v+"\n")
        f.write('*****************************************************************\n')
        f.close()
        time.sleep(60)
nvidia_go()

1.1-2021.01.04 版本源代码如下：

#!/usr/bin/python3
#!/usr/bin/python3
import os
import datetime
import time
import re

filepath = '/root/nvidia/'

def nvidia_go():
    #f=open(filepath,'a')
    #f.write("Sleep 180s, Please wait...\n")
    #f.close()
    #time.sleep(180)
    while True:
        f_name = datetime.datetime.now().strftime('%Y%m%d')
        file_log = filepath+f_name
        f = open(file_log,'a')
        time_val = datetime.datetime.now().strftime('%Y-%m-%d %H:%M:%S')
        f.write('********************************************************************************\n')
        f.write(time_val+"\n")
        val = os.popen("nvidia-smi")
        while not val:
            f.write("Please wait soft start...\n")
            time.sleep(1)
        for v in val:
            f.write(v+"\n")
            result = re.search("GPU is lost",v)
            if result:
                r = open("/root/nvidia/reboot.log",'a')
                r.write('********************************************************************************\n')
                r.write(time_val+"\n")
                r.write("GPU is lost, Server will be reboot!\n")
                r.close()
                f.close()
                os.popen("reboot")
        f.close()
        time.sleep(60)
nvidia_go()

开启开机自启：
1、建立 rc-local.service 文件
sudo vi /etc/systemd/system/rc-local.service
2、将下列内容替换 rc-local.service 文件

[Unit]
Description=/etc/rc.local Compatibility
ConditionPathExists=/etc/rc.local
 
[Service]
Type=forking
ExecStart=/etc/rc.local start
TimeoutSec=0
StandardOutput=tty
RemainAfterExit=yes
SysVStartPriority=99
 
[Install]
WantedBy=multi-user.target

3、创建文件 rc.local
sudo vi /etc/rc.local

4、将下列内容复制进 rc.local 文件

#!/bin/sh -e
# rc.local
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will "exit 0" on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#
# By default this script does nothing.
/root/nvidia.sh>/root/nvidia.sh.log 2>&1
exit 0

5、给 rc.local 加上权限
sudo chmod +x /etc/rc.local
6、启用服务
sudo systemctl enable rc-local
7、启动服务并检查状态
sudo systemctl start rc-local.service
sudo systemctl status rc-local.service
8、创建编辑 nvidia.sh 文件

#！/bin/bash
#nvidia.sh
/usr/bin/nohup /usr/bin/python3 -u /root/nvidia.py > /root/nvidia.txt 2>&1 &