Go运维开发之日志收集（8）将应用程序日志写入到文件中

Go运维开发之日志收集（1）收集应用程序日志到kafka中

Go运维开发之日志收集（2）从etcd中获取配置信息

Go运维开发之日志收集（3）根据etcd配置项创建多个tailTask

Go运维开发之日志收集（4）监视etcd配置项的变更

Go运维开发之日志收集（5）根据IP地址去拉取配置日志收集项

Go运维开发之日志收集（6）从kafka中获取日志信息

Go运维开发之日志收集（7）将日志入库到Elasticsearch并通过Kibana进行展示

前面已经开发了logAgent和logTransfer两个应用程序，日志直接使用的fmt输出的。很多时候程序都是在后台运行的，出了问题的时候才需要上去查看日志定位问题，这就需要将我们的日志保存到文件中来。

使用logrus记录日志

常用的第三方日志库有logrus,zap等，基础用法可以查看官方的页面或者我前面的文章：

Go语言第三方日志库logrus使用

Go语言高性能日志库zap使用

这里使用的是logrus来记录日志

安装logrus

1go get github.com/sirupsen/logrus

自定义logger

修改logAgent/conf/config.ini文件，添加log节相关信息

1[log]
2filename = my.log
3loglevel = debug

修改对应的结构体文件logCollect/conf/config.go

 1type AppConf struct {
 2	CenterConf `ini:"center"`
 3	KafkaConf  `ini:"kafka"`
 4	EtcdConf   `ini:"etcd"`
 5	LogConf    `ini:"log"`
 6}
 7
 8type LogConf struct {
 9	FileName string `ini:"filename"`
10	LogLevel string `ini:"loglevel"`
11}

创建logCollect/logAgent/logger目录并创建logger.go文件

 1package logger
 2
 3import (
 4	"os"
 5
 6	"github.com/sirupsen/logrus"
 7)
 8
 9var Log = logrus.New()
10
11func Init(filepath, loglevel string) (err error) {
12	// The API for setting attributes is a little different than the package level
13	// exported logger. See Godoc.
14	Log.SetFormatter(&logrus.JSONFormatter{})
15	var level logrus.Level
16  // 其中日志级别
17	switch loglevel {
18	case "trace":
19		level = logrus.TraceLevel
20	case "debug":
21		level = logrus.DebugLevel
22	case "info":
23		level = logrus.InfoLevel
24	case "warning":
25		level = logrus.WarnLevel
26	case "error":
27		level = logrus.ErrorLevel
28	case "fatal":
29		level = logrus.FatalLevel
30	case "panic":
31		level = logrus.PanicLevel
32	default:
33		level = logrus.InfoLevel
34	}
35	Log.SetLevel(level)
36	//Log.Out = os.Stdout
37
38	// You could set this to any `io.Writer` such as a file
39	file, err := os.OpenFile(filepath, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
40	if err != nil {
41		Log.Out = os.Stdout
42		return
43	}
44	Log.Out = file
45	return
46}

main.go函数调用logger进行初始化，放在加载完配置文件之后

1  // 加载日志文件
2	err = logger.Init(cfg.LogConf.FileName, cfg.LogConf.LogLevel)
3	if err != nil {
4		logger.Log.Warnf("初始化日志文件失败, err:%v\n", err)
5	}
6	logger.Log.Info("初始化日志文件成功")

修改应用程序的其他页面，将fmt.Printf,fmt.Println等修改为logger.Log.Infof之类的级别。

构建测试

1go build
2./logAgent
32020/03/09 18:12:57 Seeked /private/var/log/system.log - &{Offset:0 Whence:2}

检查日志文件

1{"level":"info","msg":"初始化日志文件成功","time":"2020-03-09T18:12:57+08:00"}
2{"level":"info","msg":"init kafka success","time":"2020-03-09T18:12:57+08:00"}
3{"level":"info","msg":"init etcd success.","time":"2020-03-09T18:12:57+08:00"}
4{"level":"info","msg":"get conf from etcd success, [0xc0002862c0]\n","time":"2020-03-09T18:12:57+08:00"}
5{"level":"debug","msg":"index:0 value:\u0026{/private/var/log/system.log system_log}\n","time":"2020-03-09T18:12:57+08:00"}
6{"level":"debug","msg":"get log data from /private/var/log/system.log success, log:Mar  9 18:12:58 huangzhgdedeAir com.apple.xpc.launchd[1] (com.oray.sunlogin.desktopagent[86984]): Service could not initialize: Unable to set current working directory. error = 2: No such file or directory, path = /Applications/SunloginClient.app/Contents/Helpers: 19D76: xpcproxy + 14893 [673][F168B004-7C3C-33AD-8EA9-EF5E7F784A63]: 0x2\n","time":"2020-03-09T18:12:58+08:00"}
7{"level":"debug","msg":"get log data from /private/var/log/system.log success, log:Mar  9 18:12:58 huangzhgdedeAir com.apple.xpc.launchd[1] (com.oray.sunlogin.desktopagent[86984]): Service exited with abnormal code: 78\n","time":"2020-03-09T18:12:58+08:00"}
8{"level":"debug","msg":"get log data from /private/var/log/system.log success, log:Mar  9 18:12:58 huangzhgdedeAir com.apple.xpc.launchd[1] (com.oray.sunlogin.desktopagent): Service only ran for 0 seconds. Pushing respawn out by 10 seconds.\n","time":"2020-03-09T18:12:58+08:00"}

看到文件已经写入到文件中了。

设置日志文件切割

logrus默认不带日志切割功能的，我们使用第三方库

github.com/lestrrat-go/file-rotatelogs 和 github.com/rifflock/lfshook

修改配置文件conf/config.ini加入日志切割相关的参数

1[log]
2# 日志文件目录
3filePath = logs
4# 日志文件名称
5filename = my.log
6# 日志级别 trace,debug,info,warning,error,fatal,panic
7loglevel = debug
8# 最长保存多少天
9max_age = 7

修改config.go结构体文件

1type LogConf struct {
2	FilePath string `ini:"filePath"`
3	FileName string `ini:"filename"`
4	LogLevel string `ini:"loglevel"`
5	MaxAge   int    `ini:"max_age"`
6}

修改logger.go文件

 1package logger
 2
 3import (
 4	"os"
 5	"path"
 6	"time"
 7
 8	rotatelogs "github.com/lestrrat-go/file-rotatelogs"
 9	"github.com/rifflock/lfshook"
10	"github.com/sirupsen/logrus"
11)
12
13var Log = logrus.New()
14
15func Init(logfile, loglevel string, maxAge time.Duration) (err error) {
16	// The API for setting attributes is a little different than the package level
17	// exported logger. See Godoc.
18	Log.Out = os.Stdout
19	// 判断目录是否存在，不存在则创建
20	dir := path.Dir(logfile)
21	if _, err = os.Stat(dir); os.IsNotExist(err) {
22		err = os.MkdirAll(dir, 0755)
23		if err != nil {
24			Log.Warnf("create directory failed, err:%v\n", err)
25		}
26	}
27
28	// 设置日志输出格式
29	Log.SetFormatter(&logrus.JSONFormatter{})
30	var level logrus.Level
31	switch loglevel {
32	case "trace":
33		level = logrus.TraceLevel
34	case "debug":
35		level = logrus.DebugLevel
36	case "info":
37		level = logrus.InfoLevel
38	case "warning":
39		level = logrus.WarnLevel
40	case "error":
41		level = logrus.ErrorLevel
42	case "fatal":
43		level = logrus.FatalLevel
44	case "panic":
45		level = logrus.PanicLevel
46	default:
47		level = logrus.InfoLevel
48	}
49	Log.SetLevel(level)
50	//Log.Out = os.Stdout
51
52	writer, err := rotatelogs.New(
53		// 分割后的文件名称
54		logfile+".%Y%m%d.log",
55		// 生成软链，指向最新日志文件
56		rotatelogs.WithLinkName(logfile),
57		// 设置最大保存时间(7天)
58		rotatelogs.WithMaxAge(maxAge),
59		// 设置日志切割时间间隔(1天)
60		rotatelogs.WithRotationTime(24*time.Hour),
61	)
62
63	writeMap := lfshook.WriterMap{
64		logrus.InfoLevel:  writer,
65		logrus.FatalLevel: writer,
66		logrus.DebugLevel: writer,
67		logrus.WarnLevel:  writer,
68		logrus.ErrorLevel: writer,
69		logrus.PanicLevel: writer,
70	}
71
72	lfHook := lfshook.NewHook(writeMap, &logrus.JSONFormatter{
73		TimestampFormat: "2006-01-02 15:04:05",
74	})
75
76	// 新增 Hook
77	Log.AddHook(lfHook)
78
79	// You could set this to any `io.Writer` such as a file
80	file, err := os.OpenFile(logfile, os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
81	if err != nil {
82		return
83	}
84	Log.Out = file
85	return
86}

修改main函数入口

1	err = logger.Init(path.Join(cfg.LogConf.FilePath, cfg.LogConf.FileName), cfg.LogConf.LogLevel, time.Duration(cfg.LogConf.MaxAge)*time.Hour*24)
2	if err != nil {
3		logger.Log.Warnf("初始化日志文件失败, err:%v\n", err)
4	}

构建测试

1go build
2./logAgent

查看生成的日志文件

1ls -lh logs
2total 8
3lrwxr-xr-x  1 jerry  staff    24B  3  9 20:18 my.log -> logs/my.log.20200309.log
4-rw-r--r--  1 jerry  staff   3.9K  3  9 20:18 my.log.20200309.log

已经生成软链接并指向我们配置的文件名。

修改logTransfer

使用同样的方法去修改logTransfer，这里就不一一介绍了，具体可以查看Github-logCollect