当前位置：首页 > web >正文

Loki日志体系的搭建

web 2025/5/6 14:37:55

前言

日志做为运维监控体系重要的组成部分，其重要性已毋容置疑。ELK做为老牌日志解决方案，几乎成为了日志系统的代名词，但ELK却存在以下问题：

系统太重，因为是JAVA开发的，ES就是耗内存的代名词
配置太复杂，需要一定的经验才能驾驭
无法做到服务粒度的方便查询

而Grafana技术体系的Loki，则完美解决了以上问题：

因为是用Go写的，内存占用差不多少是ES的十分之一
基于Prometheus格式，使用标签对日志内容进行分类，既提升了查询速度，又可以对不同服务不同环境进行细粒度的查询
配置简单，无需对各种格式的日志进行二次过滤

安装

Loki日志体系主要分三个组件：

Loki，日志存储
Promtail，日志采集
Grafana，日志查询

Loki安装

创建配置文件local-config.yaml，内容如下：

auth_enabled: false

server:
http_listen_port: 3100

common:
instance_addr: 127.0.0.1
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory

query_range:
results_cache:
cache:
embedded_cache:
enabled: true
max_size_mb: 100

schema_config:
configs:
- from: 2020-10-24
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
storage_config:
tsdb_shipper:
active_index_directory: /loki/index # 索引存储目录
cache_location: /loki/index_cache # 索引缓存目录
filesystem: # 使用文件系统作为对象存储
directory: /loki/chunks # Loki 数据块存储路径，替换成你自己的存储路径

limits_config:
max_query_lookback: 72h # 28 天
retention_period: 72h # 28 天

compactor:
working_directory: /loki/retention
delete_request_store: filesystem # 将存储请求的删除存储方式设置为文件系统
retention_enabled: true

K8S中部署Loki，记得要先创建存储卷，以便持久化使用，Yaml配置内容参考如下：

apiVersion: apps/v1
kind: StatefulSet
metadata:
labels:
app: loki
name: loki
namespace: devops
spec:
selector:
matchLabels:
app: loki
template:
metadata:
labels:
app: loki
spec:
containers:
- env:
- name: TZ
value: Asia/Shanghai
image: grafana/loki:3.4
name: loki
resources:
limits:
cpu: "2"
memory: 4Gi
requests:
cpu: "1"
memory: 2Gi
volumeMounts:
- mountPath: /loki
name: loki-data
- mountPath: /etc/loki
name: loki-config
volumes:
- name: loki-data
persistentVolumeClaim:
claimName: loki-data
- configMap:
defaultMode: 420
name: loki-config
name: loki-config

Promtail安装

K8S部署方式

helm repo add grafana https://grafana.github.io/helm-charts
helm repo update
helm install promtail grafana/promtail -n devops

Linux单机部署方式

curl -O -L "https://github.com/grafana/loki/releases/download/v3.4.3/promtail-linux-amd64.zip"
unzip promtail-linux-amd64.zip
chmod a+x promtail-linux-amd64
cp promtail-linux-amd64 /usr/local/bin/promtail

vim /etc/systemd/system/promtail.service
#粘贴以下内容
[Unit]
Description=Promtail service
After=network.target

[Service]
Type=simple
User=root
ExecStart=/usr/local/bin/promtail -config.file /etc/promtail/promtail-config.yaml
Restart=on-failure
RestartSec=20
StandardOutput=append:/var/log/promtail.log
StandardError=append:/var/log/promtail.log

[Install]
WantedBy=multi-user.target

systemctl daemon-reload
systemctl enable promtail
systemctl start promtail

K8S中的采集配置文件promtail.yaml，内容参考：

server:
log_level: info
log_format: logfmt
http_listen_port: 3101

clients:
- url: http://loki.devops.svc.cluster.local:3100/loki/api/v1/push

positions:
filename: /run/promtail/positions.yaml

scrape_configs:

# 通过 kubernetes_sd_configs:pod 配置 pod 日志，参考 https://grafana.com/docs/loki/latest/clients/promtail/configuration/#kubernetes_sd_config
- job_name: kubernetes-pods-app
# 流水线
pipeline_stages:
- docker: {}
kubernetes_sd_configs:
- role: pod
relabel_configs:
# 把 pod 所有的标签暴露出来
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)
replacement: $1
target_label: $1
- action: drop
regex: .+
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- action: replace
source_labels:
- __meta_kubernetes_pod_ip
target_label: pod_ip
- action: replace
source_labels:
- __meta_kubernetes_pod_label_app
target_label: app
- action: drop
regex: ''
source_labels:
- app
- action: replace
source_labels:
- __meta_kubernetes_pod_label_component
target_label: component
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: node_name
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
- action: replace
replacement: $1
separator: /
source_labels:
- namespace
- app
target_label: job
- action: replace
source_labels:
- __meta_kubernetes_pod_name
target_label: pod
- action: replace
source_labels:
- __meta_kubernetes_pod_container_name
target_label: container
- action: replace
replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_uid
- __meta_kubernetes_pod_container_name
target_label: __path__
- action: replace
regex: true/(.*)
replacement: /var/log/pods/*$1/*.log
separator: /
source_labels:
- __meta_kubernetes_pod_annotationpresent_kubernetes_io_config_hash
- __meta_kubernetes_pod_annotation_kubernetes_io_config_hash
- __meta_kubernetes_pod_container_name
target_label: __path__


limits_config:


tracing:
enabled: false

Linux单机日志采集文件promtail-conf.yaml文件内容参考：

server:
http_listen_port: 9080
grpc_listen_port: 0

positions:
filename: /tmp/positions.yaml

clients:
- url: 'http://IP:3100/loki/api/v1/push'

scrape_configs:
- job_name: xxxx
static_configs:
- targets:
- localhost
labels:
job: error
app: vop-proxy
namespace: prod
__path__: /app/logs/xxx-error.log
- targets:
- localhost
labels:
job: info
app: vop-proxy
namespace: prod
__path__: /app/logs/xxx-info.log