DuckDB实时日志分析系统:从Nginx日志到可视化看板

项目简介

传统日志分析需要ELK这么重的方案。用DuckDB可以直接对Nginx日志跑SQL分析,轻量级替代方案。

核心配置

# 安装DuckDB
curl -fsSL https://install.duckdb.org | sh

# 将Nginx日志导入DuckDB
duckdb -c "
CREATE TABLE logs AS 
SELECT * FROM read_csv_auto('/var/log/nginx/access.log');
"

SQL分析示例

-- 查询Top 10访问IP
SELECT client_ip, COUNT(*) as visits
FROM logs
GROUP BY client_ip
ORDER BY visits DESC
LIMIT 10;

-- 查询404错误最多的URL
SELECT request_path, COUNT(*) as errors
FROM logs
WHERE status = 404
GROUP BY request_path
ORDER BY errors DESC;

自动报告

配置cron每天生成分析报告推送微信,替代人工查看日志。

这篇文章对你有帮助吗?