python实时监控logstash日志代码
作者:hey,emmby 发布时间:2021-03-08 07:46:49
标签:python,监控,logstash,日志
实时读取logstash日志,有异常错误keywork即触发报警。
# /usr/bin/env python3
# -*- coding: utf-8 -*-
# __author__ = caozhi
# create_time 2018-11-12,update_time 2018-11-15
# version = 1.0
# 录像高可用报警
# 1 读取日志 使用游标移动
# 2 线上业务日志文件会切割,切割后,读取上一个切割的日志
import os
import sys
import json
import requests
import time
import re
cini = conf.ini'
log_file = logstash.log'
def readconf():
try:
with open(cini, 'r+') as f:
CONF = json.load(f)
except:
CONF = {"seek": 0, "inode": 922817, "last_file": logstash.log"}
writeconf(CONF=CONF)
print('conf.ini 配置文件缺失,自动创建一个新的配置文件')
return CONF
def writeconf(CONF):
with open(cini, 'w+') as e:
json.dump(CONF, e)
def read_log(log_file, seek):
try:
f = open(log_file, 'r')
except FileNotFoundError:
f = open(logstash.log', 'r')
seek = 0
print('上一个文件读取失败了,请检查切割的日志文件')
except:
print('日志文件打开错误,退出程序')
sys.exit()
f.seek(seek)
line = f.readline()
new_seek = f.tell()
if new_seek == seek:
print('没有追加日志,退出程序')
sys.exit()
while line:
try:
logstash = json.loads(line)
except:
CONF = {"seek": 0, "inode": 922817, "last_file": "/data/logs/lmrs/logstash.log"}
writeconf(CONF=CONF)
print('json数据加载错误,重新创建一个新的配置文件')
sys.exit()
#if '''re.search(time.strftime("%Y:%H:%M", time.localtime()), logstash.get('log_time')) and '''logstash.get('rtype') == 6 and logstash.get('uri') == '/publish' and logstash.get('event') == 0:
if logstash.get('rtype') == 6 and logstash.get('uri') == '/publish' and logstash.get('event') == 0:
value = 1
stream = logstash.get('name')
print('{} {}'.format(value, stream))
record(value=value, stream=stream)
else:
value = 0
stream = 0
line = f.readline()
seek = f.tell()
f.close
return value, stream, seek
def record(value, stream):
data = []
record = {}
record['metric'] = 'recording_high_availability_monitor'
record['endpoint'] = os.uname()[1]
record['timestamp'] = int(time.time())
record['step'] = 60
record['value'] = value
record['counterType'] = 'GAUGE'
record['Tags'] = '{}={}'.format(int(time.time()), stream)
data.append(record)
if data:
print('这是data的json数据')
print(data)
falcon_request = requests.post("http://127.0.0.1:1988/v1/push", data=json.dumps(data))
#falcon_request = requests.post("http://127.0.0.1:1988/v1/push", json=data)
print('json参数请求返回状态码为:' + str(falcon_request.status_code))
print('json参数请求返回为:' + str(falcon_request.text))
if __name__ == '__main__':
print()
print('***************************************')
print('本次执行脚本时间:{}'.format(time.strftime("%Y%m%d_%H%M", time.localtime())))
CONF = readconf()
print('first_CONF :{}'.format(CONF))
print('NO1.log_file',log_file)
last_inode = CONF['inode']
inode = os.stat(log_file).st_ino
print('last_inode: {} inode: {}'.format(last_inode, inode))
if inode == last_inode:
seek = CONF['seek']
next_file = 0
else:
log_file = CONF['last_file'] + time.strftime("-%Y%m%d_", time.localtime()) + str(time.strftime("%H%M", time.localtime()))[:-1] + '0'
next_file = 1
seek = CONF['seek']
print('NO2.log_file',log_file)
value, stream, seek = read_log(log_file=log_file,seek=seek)
if next_file:
CONF['seek'] = 0
else:
CONF['seek'] = seek
CONF['inode'] = os.stat(logstash.log').st_ino
writeconf(CONF=CONF)
print('last_CONF :{}'.format(CONF))
补充知识:logstash 调用exec
我就废话不多说了,还是直接看代码吧!
[elk@Vsftp logstash]$ cat t3.conf
input {
stdin {
}
}
filter {
grok {
match => [ "message","(?m)\s*%{TIMESTAMP_ISO8601:time}\s*(?<Level>(\S+)).*"]
}
date {
match => ["time", "yyyy-MM-dd HH:mm:ss,SSS"]
}
mutate {
add_field =>["type","tailong"]
add_field =>["messager","%{type}-%{message}"]
remove_field =>["message"]
}
}
output {
if ([Level] == "ERROR" or [messager] =~ "Exception" ) and [messager] !~ "温金服务未连接" and [messager] !~ "调用温金代理系统接口错误" and [messager] !~ "BusinessException" {
exec {
command => "/bin/smail.pl \"%{messager}\" \"%{type}\" "
}
}
stdout {
codec =>rubydebug
}
}
Vsftp:/root# cat /bin/smail.pl
#!/usr/bin/perl
use Net::SMTP;
use HTTP::Date qw(time2iso str2time time2iso time2isoz);
use Data::Dumper;
use Getopt::Std;
use vars qw($opt_d );
getopts('d:');
# mail_user should be your_mail@163.com
$message= "@ARGV";
$env="$opt_d";
sub send_mail{
my $CurrTime = time2iso(time());
my $to_address = shift;
my $mail_user = 'zhao.yangjian@163.com';
my $mail_pwd = 'xx';
my $mail_server = 'smtp.163.com';
my $from = "From: $mail_user\n";
my $subject = "Subject: zjcap info\n";
my $info = "$CurrTime--$message";
my $message = <<CONTENT;
$info
CONTENT
my $smtp = Net::SMTP->new($mail_server);
$smtp->auth($mail_user, $mail_pwd) || die "Auth Error! $!";
$smtp->mail($mail_user);
$smtp->to($to_address);
$smtp->data(); # begin the data
$smtp->datasend($from); # set user
$smtp->datasend($subject); # set subject
$smtp->datasend("\n\n");
$smtp->datasend("$message\n"); # set content
$smtp->dataend();
$smtp->quit();
};
send_mail ('zhao.yangjian@163.com');
2017-01-12 10:19:19,888 jjjjj Exception
{
"@version" => "1",
"@timestamp" => "2017-01-12T02:19:19.888Z",
"host" => "Vsftp",
"time" => "2017-01-12 10:19:19,888",
"Level" => "jjjjj",
"type" => "tailong",
"messager" => "tailong-2017-01-12 10:19:19,888 jjjjj Exception"
}
来源:https://blog.csdn.net/xiaozhiit/article/details/84431541


猜你喜欢
- 最近在公司接到一个需求,里面有一个 * 跳转。类似于选择地址的时候,选择的顺序是:省份->市->区。如果分三个页面跳转,那么体验非
- TensorFlow训练时,遇到内存不断增长,最终导致内存不足,进程被杀死。在这里我不准备对造成这一现象的所有原因进行探讨,只是记录一下我在
- 菜鸟一枚,写着试了试,虽说有点杂乱,但还是能用,我是在linux下运行的大致说下过程:1、把需要ping的网段中所有ip存到数组中(我是放到
- 功能:实现网页内容的即时编辑,增加页面的可用性、交互性。方法1:直接通过textarea标签实现,请运行下边代码:<!DOCTYPE
- 创建一张作者表,author,创建外键与book表,多对多关系 ,外键字段放在那张表都可以,class Author(models.Mode
- 1. 简介NumPy(Numerical Python) 是 Python 语言的一个扩展程序库,支持大量的维度数组与矩阵运算,此外也针对数
- 不知大家对精华区的表格排序终极优化是否还有记忆,当时讨论的结果曾以为是最快的JS排序了,实则不然,按前段时间我发的DHTML性能提升帖(转译
- js实现点击掉落特效 先看看效果图 话不多说代码<!DOCTYPE HTML><html><head
- 单继承时super()和__init__()实现的功能是类似的class Base(object):def __init__(self):p
- MySQL在5.1引入了一个rename database操作,但在MySQL5.1.23后又不支持这个命令。可以说是一个实验性的功能,没有
- 本文研究的主要是Django使用httpresponse返回用户头像,下面是相关实例代码。当请求一个页面时,Django 把请求的 meta
- 目录问题复现隐式转换总结参考问题在工作中发现,有一个接口只执行一条SQL查询语句,并且SQL明明使用了主键列,但是速度很慢。在MySQL中E
- 谷歌驱动下载地址:http://chromedriver.storage.googleapis.com/index.html一、seleni
- value_counts()方法返回一个序列Series,该序列包含每个值的数量(对于数据框中的任何列,value_counts()方法会返
- 什么是事务事务就是一组操作的集合,事务将整组操作作为一个整体,共同提交或者共同撤销这些操作只能同时成功或者同时失败,成功即可提交事务,失败就
- 废话不多说,直接开始拉~~~我们总共有 6 只海龟,颜色不同,它们以随机长度移动。首先,我们应该通过输入乌龟的颜色来押注乌龟。第一个越线的乌
- 目录一、前言二、使用1.@Component2.compued、data、methods3.@props4.@watch5.@emit三 、
- 今天摸了半天摸出来的,虽然这里没啥人玩ASP,不过也是win7的问题,发发当备份问题1.An error occurre
- Python产生一个数值范围内的不重复的随机数,可以使用random模块中的random.sample函数,其用法如下:import ran
- 在Python中,实现循环语句有以下几种方式:1. for 循环for 循环是 Python 中最常用的循环语句之一,可以遍历任何序列,如一