python 日志增量抓取实现方法
2019/7/15 0:50:50
本文主要是介绍python 日志增量抓取实现方法,对大家解决编程问题具有一定的参考价值,需要的程序猿们随着小编来一起学习吧!
实例如下所示:
import time import pickle import os import re class LogIncScaner(object): def __init__(self,log_file, reg_ex,seek_file='/tmp/log-inc-scan.seek.temp'): self.log_file = log_file self.reg_ex = reg_ex self.seek_file = seek_file def scan(self): seek = self._get_seek() file_mtime = os.path.getmtime(self.log_file) if file_mtime <= seek['time']: print 'file mtime not change since last scan' seek['time'] = file_mtime self._dump_seek(seek) return [] file_size = os.path.getsize(self.log_file) if file_size <= seek['position']: print 'file size not change since last scan' seek['position'] = file_size self._dump_seek(seek) return [] print 'file changed,start to scan' matchs = [] with open(self.log_file, 'rb') as logfd: logfd.seek(seek['position'],os.SEEK_SET) for match in re.finditer(self.reg_ex, logfd.read()): matchs.append(match) seek = {'time':time.time(),'position': logfd.tell()} print seek self._dump_seek(seek) return matchs def _get_seek(self): seek = {'time':time.time(),'position':0} if os.path.exists(self.seek_file): with open(self.seek_file,'rb') as seekfd: try: seek = pickle.load(seekfd) except: pass print seek return seek def _dump_seek(self, seek): with open(self.seek_file,'wb') as seekfd: pickle.dump(seek,seekfd) def reset_seek(self): self._dump_seek({'time':time.time(),'position':0}) if __name__ == "__main__": scaner = LogIncScaner('/var/log/messages',r'(\w+ \d+ \d+:\d+:\d+) .+?exception') scaner.reset_seek() while True: matchs = scaner.scan() for match in matchs: print 'fond at:' + match.group(1) + ' content:' + match.group(0) time.sleep(5)
以上这篇python 日志增量抓取实现方法就是小编分享给大家的全部内容了,希望能给大家一个参考,也希望大家多多支持找一找教程网。
这篇关于python 日志增量抓取实现方法的文章就介绍到这儿,希望我们推荐的文章对大家有所帮助,也希望大家多多支持为之网!
- 2024-09-27使用python 将ETH账户的资产打散
- 2024-09-26Python编程基础
- 2024-09-2610 种方法写出更好的 Python 代码
- 2024-09-25Python编程基础详解
- 2024-09-25Python编程入门教程
- 2024-09-25从零开始使用Python构建LLaMA 3
- 2024-09-23Python中理解和使用树形结构的简单教程
- 2024-09-23Python 编程基础入门
- 2024-09-18初探Python股票自动化交易:入门指南
- 2024-09-18Python量化入门:轻松掌握量化分析基础与实战