当前位置： 首页 > backend >正文

【APM】Build an environment for Traces, Metrics and Logs of App by OpenTelemetry

backend 2025/5/22 10:54:02

系列文章目录

此系列文章介绍如何搭建Observability（可观测性）环境(Opentelemetry-Collector、Tempo、Prometheus、Loki和Grafana)，以及应用。

【APM】Observability Solution

【APM】Build an environment for Traces, Metrics and Logs of App by OpenTelemetry

【APM】NET Traces, Metrics and Logs to OLTP

【APM】How to enable Trace to Logs on Grafana?

前言

在现代化的分布式应用程序中，Observability（可观测性）对于监控、排查故障和优化性能至关重要。本系列文章显示如何整合OpenTelemetry、Opentelemetry-Collector、Tempo、Prometheus、Loki和Grafana来提升.NET应用程序的可观测性，以及使用Promtail收集Nginx access log，.NET应用程序通过Serilog 记录日志到Loki。

一、先决条件

在开始之前，确保你已经安装以下工具：

  • Docker
  • .NET SDK

二、Docker image download and re-tag

1.Download images

There are 8 images of docker that need to be downloaded.

  1. otel/opentelemetry-collector-contrib
  2. otel/opentelemetry-collector (not support exporter logs to loki, replaced by otel/opentelemetry-collector-contrib)
  3. grafana/tempo
  4. grafana/loki
  5. prom/prometheus
  6. grafana/grafana
  7. grafana/promtail
  8. your/nginx

The scripts as below:

docker pull  otel/opentelemetry-collector-contrib:latest
docker pull  otel/opentelemetry-collector:latest
docker pull  grafana/tempo:latest
docker pull  grafana/loki:latest
docker pull  prom/prometheus:latest
docker pull  grafana/grafana:latest
docker pull  grafana/promtail:latest
docker pull  your/nginx:latest

2.Get versions for each images 

Use docker run, the sample code as below:

docker rm -f grafana
docker run -it -p 3000:3000 --rm -e http_proxy="" --name grafana grafana:latest# If you can't go to the container, please use the command: docker logs grafana, then get a version from the logs.
docker logs grafana...

3.Re-tag all images

The scripts as below:

# 当使用latest tag，会出现兼容性问题，故以程式的版本作为tag。
docker tag otel/opentelemetry-collector-contrib:latest otel/opentelemetry-collector-contrib:0.123.0
docker tag otel/opentelemetry-collector:latest otel/opentelemetry-collector:0.123.0
docker tag grafana/tempo:latest grafana/tempo:2.7.2
docker tag grafana/loki:latest grafana/loki:3.4.3
docker tag prom/prometheus:latest prom/prometheus:3.3.0
docker tag grafana/grafana:latest grafana/grafana:11.1.0
docker tag grafana/promtail:latest grafana/promtail:3.4.2
docker tag your/nginx:latest your/nginx:1.0.11# delete original images
docker rmi otel/opentelemetry-collector-contrib:latest
docker rmi otel/opentelemetry-collector:latest
docker rmi grafana/tempo:latest
docker rmi grafana/loki:latest
docker rmi prom/prometheus:latest
docker rmi grafana/grafana:latest
docker rmi grafana/promtail:latest
docker rmi your/nginx:latest

二、Setting configuration

1.loki.yaml


# This is a complete configuration to deploy Loki backed by the filesystem.
# The index will be shipped to the storage via tsdb-shipper.auth_enabled: falseserver:http_listen_port: 3100http_listen_address: 0.0.0.0  # 监听所有 IPv4 地址common:ring:instance_addr: 127.0.0.1kvstore:store: inmemoryreplication_factor: 1path_prefix: /tmp/lokischema_config:configs:- from: 2020-05-15store: tsdbobject_store: filesystemschema: v13index:prefix: index_period: 24hstorage_config:filesystem:directory: /tmp/loki/chunks

2.otel-collector.yaml

receivers:otlp:protocols:grpc:endpoint: 0.0.0.0:4317http:endpoint: 0.0.0.0:4318exporters:debug:verbosity: detailedprometheus:endpoint: ":9201"send_timestamps: truemetric_expiration: 180menable_open_metrics: trueotlp:endpoint: tempo:4317tls:insecure: true# 转发 Logs 到 Lokiloki:endpoint: "http://loki:3100/loki/api/v1/push"tls:insecure: truedefault_labels_enabled:exporter: truejob: true service:pipelines:traces:receivers: [otlp]exporters: [debug, otlp]metrics:receivers: [otlp]exporters: [debug, prometheus]logs:receivers: [otlp]exporters: [debug, loki]

3.prometheus.yaml

global:scrape_interval:     15sevaluation_interval: 15sscrape_configs:- job_name: 'otel'static_configs:- targets: [ 'otel-collector:9201' ]

4.promtail-config.yaml

# Promtail服务配置
server:http_listen_port: 9080grpc_listen_port: 0# 记录读取日志的位置信息文件,Promtail重新启动时需要它
positions:filename: /run/promtail/positions.yaml# Loki的api服务的地址
clients:- url: http://loki:3100/loki/api/v1/pushscrape_configs:
# nginx日志收集并打标签- job_name: nginx_logs # 服务名称static_configs:- targets:- localhost # 目标服务器名称labels:job: nginx # 作业名称host: localhost # 服务器地址__path__: /var/log/nginx/*.log #服务器日志路径

5.tempo.yaml

server:http_listen_port: 3200distributor:receivers:otlp:protocols:http:endpoint: "0.0.0.0:4318"grpc:endpoint: "0.0.0.0:4317"storage:trace:backend: localwal:path: /tmp/tempo/wallocal:path: /tmp/tempo/blocks

6.grafana-datasources.yaml

apiVersion: 1datasources:
- name: Prometheustype: prometheusuid: prometheusaccess: proxyorgId: 1url: http://prometheus:9090basicAuth: falseisDefault: trueversion: 1editable: falsejsonData:httpMethod: GETexemplarTraceIdDestinations:- name: trace_iddatasourceUid: Tempo
- name: Tempotype: tempoaccess: proxyorgId: 1url: http://tempo:3200basicAuth: falseisDefault: falseversion: 1editable: falseapiVersion: 1uid: tempojsonData:httpMethod: GETserviceMap:datasourceUid: prometheus

 7.nginx-custom.conf

# user  myuser;worker_processes auto;  # 自动计算可用CPU核数来决定worker进程的数量
pid /tmp/nginx.pid;     # 将PID文件放到/tmp下避免权限问题events {worker_connections 1024;  # 最大连接数
}http {include       /etc/nginx/mime.types;  # 默认MIME类型配置default_type  application/octet-stream;sendfile        on;  # 打开高效传输模式keepalive_timeout  65;  # 长连接保持时间log_format json_analytics escape=json '{''"msec": "$msec", ' # request unixtime in seconds with a milliseconds resolution'"connection": "$connection", ' # connection serial number'"connection_requests": "$connection_requests", ' # number of requests made in connection'"pid": "$pid", ' # process pid'"request_id": "$request_id", ' # the unique request id'"request_length": "$request_length", ' # request length (including headers and body)'"remote_addr": "$remote_addr", ' # client IP'"remote_user": "$remote_user", ' # client HTTP username'"remote_port": "$remote_port", ' # client port'"time_local": "$time_local", ''"time_iso8601": "$time_iso8601", ' # local time in the ISO 8601 standard format'"request": "$request", ' # full path no arguments if the request'"request_uri": "$request_uri", ' # full path and arguments if the request'"args": "$args", ' # args'"status": "$status", ' # response status code'"body_bytes_sent": "$body_bytes_sent", ' # the number of body bytes exclude headers sent to a client'"bytes_sent": "$bytes_sent", ' # the number of bytes sent to a client'"http_referer": "$http_referer", ' # HTTP referer'"http_user_agent": "$http_user_agent", ' # user agent'"http_x_forwarded_for": "$http_x_forwarded_for", ' # http_x_forwarded_for'"http_host": "$http_host", ' # the request Host: header'"server_name": "$server_name", ' # the name of the vhost serving the request'"request_time": "$request_time", ' # request processing time in seconds with msec resolution'"upstream": "$upstream_addr", ' # upstream backend server for proxied requests'"upstream_connect_time": "$upstream_connect_time", ' # upstream handshake time incl. TLS'"upstream_header_time": "$upstream_header_time", ' # time spent receiving upstream headers'"upstream_response_time": "$upstream_response_time", ' # time spend receiving upstream body'"upstream_response_length": "$upstream_response_length", ' # upstream response length'"upstream_cache_status": "$upstream_cache_status", ' # cache HIT/MISS where applicable'"ssl_protocol": "$ssl_protocol", ' # TLS protocol'"ssl_cipher": "$ssl_cipher", ' # TLS cipher'"scheme": "$scheme", ' # http or https'"request_method": "$request_method", ' # request method'"server_protocol": "$server_protocol", ' # request protocol, like HTTP/1.1 or HTTP/2.0'"pipe": "$pipe", ' # "p" if request was pipelined, "." otherwise'"gzip_ratio": "$gzip_ratio", ''"http_cf_ray": "$http_cf_ray",'# '"http_authorization": "$http_authorization", ' # 添加 Authorization 头信息 # added by Rubin @2025/4/6 '"geoip_country_code": ""' # "$geoip_country_code"'}';# 日志配置（按天生成日誌文件名）map $time_iso8601 $logdate {'~^(?<ymd>\d{4}-\d{2}-\d{2})' $ymd;default 'date-not-found';}# access_log /var/log/nginx/access.log;access_log /var/log/nginx/access_$logdate.log json_analytics;error_log /var/log/nginx/error.log warn;# 简单的HTTP服务器设置server {listen       80;  # 监听80端口# 请求根目录时返回的内容location / {root   /usr/share/nginx/html;  # 网页所在目录index  index.html;   # 默认主页}error_page 404  /index.html;# 错误页面设置error_page  500 502 503 504  /50x.html;location = /50x.html {root   /usr/share/nginx/html;}}# Optionally, enable gzip for all server blocksgzip  on;
}

8.docker-compose.yml

# version: "3.8"services:# OTEL Collector to receive logs, metrics and traces from the applicationotel-collector:#image: otel/opentelemetry-collector:0.123.0 # no loki exporterimage: otel/opentelemetry-collector-contrib:0.123.0container_name: otel-collectorcommand: [ "--config=/etc/otel-collector.yaml" ]volumes:- ./otel-collector.yaml:/etc/otel-collector.yamlports:- "4317:4317"- "4318:4318"- "9201:9201"networks:- my_network# Exports Traces to Tempotempo:image: grafana/tempo:2.7.2container_name: tempocommand: [ "-config.file=/etc/tempo.yaml" ]volumes:- ./tempo.yaml:/etc/tempo.yaml- ./tempo-data:/tmp/tempoports:- "3200"  # tempo- "4317"  # otlp grpc- "4318"  # otlp httpnetworks:- my_network# Loki for log managementloki:image: grafana/loki:3.4.3container_name: lokicommand: [ "-config.file=/etc/loki.yaml" ]volumes:- ./loki.yaml:/etc/loki.yaml- ./loki-data:/tmp/lokiports:- "3100"networks:- my_network# Exports Metrics to Prometheusprometheus:image: prom/prometheus:3.3.0container_name: prometheuscommand:- --config.file=/etc/prometheus.yaml- --web.enable-remote-write-receiver- --enable-feature=exemplar-storagevolumes:- ./prometheus.yaml:/etc/prometheus.yaml- ./prometheus-data:/prometheusports:- "9090:9090"networks:- my_network# UI to query traces, metrics and logsgrafana:image: grafana/grafana:11.1.0container_name: grafanavolumes:- ./grafana-datasources.yaml:/etc/grafana/provisioning/datasources/datasources.yaml- ./grafana-data:/var/lib/grafanaenvironment:- GF_AUTH_ANONYMOUS_ENABLED=true- GF_AUTH_ANONYMOUS_ORG_ROLE=Editor  # Admin Viewer Editor- GF_AUTH_DISABLE_LOGIN_FORM=false # true- GF_FEATURE_TOGGLES_ENABLE=traceqlEditor- http_proxy=ports:- "3000:3000"networks:- my_networkpromtail:image: grafana/promtail:3.4.2container_name: promtailcommand: [ "-config.file=/etc/promtail/config.yaml" ]volumes:- ./promtail-config.yaml:/etc/promtail/config.yaml- ./nginx-data/logs:/var/log/nginx- ./promtail-data:/run/promtailports:- "9080:9080"networks:- my_networknginx:image: your/nginx:1.0.11container_name: nginxvolumes:- ./nginx-custom.conf:/etc/nginx/nginx.conf- ./nginx-data/logs:/var/log/nginx    # mount a folder for logs- ./nginx-data/logs:/etc/nginx/html   # ports:- "8002:80"networks:- my_networknetworks:my_network:driver: bridge

三、Docker run

1.start.sh

docker-compose up -d

 2.stop.sh

docker-compose down

 3.Run ./start.sh

四、Reference

Quick start | OpenTelemetry

https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/examples/AspNetCore/docker-compose.yaml

总结

此篇文章分享了Observability（可观测性）环境的安装过程，下一篇将分享.NET程式如何把Traces, Metrics, Logs发送给opentelemetry-collector。

http://www.xdnf.cn/news/44.html

相关文章：

  • Python网络爬虫设计（二）
  • DP 32bit位宽数据扰码实现和仿真
  • Oracle 19c部署之初始化实例(三)
  • [图论]Prim
  • docker的基础知识
  • 多模态大模型的算力需求预测：从理论FLOPs到实际集群配置（搭建算力成本评估模型的方法论）
  • 每日OJ_牛客_ruby和薯条_排序+二分/滑动窗口_C++_Java
  • 知识库Qanyting部署问题总结
  • 个人博客系统后端 - 用户信息管理功能实现指南（上）
  • Ubuntu利用docker搭建Java相关环境记录（二）
  • C++学习：六个月从基础到就业——面向对象编程：重载运算符（下）
  • 容器docker入门学习
  • ubuntu24.04离线安装deb格式的mysql-community-8.4.4
  • 【C++初阶】--- list容器功能模拟实现
  • 基于flask+vue框架的灯饰安装维修系统u49cf（程序+源码+数据库+调试部署+开发环境）带论文文档1万字以上，文末可获取，系统界面在最后面。
  • 【Unity】JSON数据的存取
  • 燕山大学计算机网络之Java实现TCP数据包结构设计与收发
  • 有什么工具可以在家连接到公司内网？局域网址提供异地公网访问的那些常用方法
  • 一台 Master 多节点玩转 Kubernetes：sealos 一键部署实践
  • MahApps.Metro：专为 WPF 应用程序设计的 UI 框架
  • 【数据结构】AVL树
  • 自动驾驶系列—GLane3D: Detecting Lanes with Graph of 3D Keypoints
  • android liveData observeForever 与 observe对比
  • CS144 Lab0实战记录：搭建网络编程基础
  • 游戏引擎学习第231天
  • 02、GPIO外设(一)：基础知识
  • Windows平台使用Docker部署Neo4j
  • 从零上手GUI Guider学习LVGL——Button
  • 【Windows本地部署n8n工作流自动平台结合内网穿透远程在线访问】
  • SAP HANA使用命令行快速导出导入