建站平台 绑定域名,电子商务建设与网站规划,国外刺绣图案设计网站,免费推广网站入口2020文章目录 1. EFK日志三件套集成1.1 核心组件1.2 部署 2. Exceptionless日志系统2.1 Exceptionless核心特性2.2 Exceptionless部署文件2.3 K8s中使用Exceptionless 1. EFK日志三件套集成
1.1 核心组件
Elasticsearch#xff08;存储#xff09;Fluentd#xff08;收集器存储Fluentd收集器Kibana数据看板 每个Kubernetes节点上可以运行一个Fluentd实例这个实例会去收集所有Pod打印出来的日志流。Fluentd可以做到无侵入的收集日志 多个Kubernetes节点组成了Kubernetes集群每个Fluentd收集到的日志都推送到Elasticsearch中。Elasticsearch可以单独部署也可以部署在Kuberbetes集群中 Kibana负责管理Elasticsearch并作为数据看板 1.2 部署
需要部署Elasticsearch、Fluentd、Kibana
Elasticsearch
apiVersion: apps/v1beta1
kind: Deployment
metadata:name: elasticsearchnamespace: defaultlabels:tag: elasticsearch
spec:replicas: 1template:metadata:labels:app: elasticsearchspec:containers:- name: elasticsearchimage: elasticsearch:7.5.2imagePullPolicy: IfNotPresentenv:- name: discovery.typevalue: single-nodevolumeMounts:- mountPath: /usr/share/elasticsearch/dataname: elasticsearch-storageports:- containerPort: 9200- containerPort: 9300resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100mmemory: 512MiterminationGracePeriodSeconds: 10volumes:- name: elasticsearch-storagehostPath:path: /d/k8s/volumes/elasticsearch/datatype: DirectoryOrCreaterestartPolicy: Always--- apiVersion: v1
kind: Service
metadata:name: elasticsearchnamespace: defaultlabels:tag: elasticsearch
spec:type: NodePortports:- nodePort: 30007port: 9200targetPort: 9200name: 9200protocol: TCP- nodePort: 30008port: 9300targetPort: 9300name: 9300protocol: TCPselector:app: elasticsearchFluentd
apiVersion: apps/v1beta1
kind: Deployment
metadata:name: fluentdnamespace: defaultlabels:tag: fluentd
spec:replicas: 1template:metadata:labels:app: fluentdspec:containers:- name: fluentdimage: witskeeper/fluentd-es:68imagePullPolicy: IfNotPresentenv:- name: discovery.typevalue: single-nodevolumeMounts:- mountPath: /fluentd/etc/fluent.confname: fluentd-configsubPath: fluent.conf- mountPath: /var/logname: containers-logs- mountPath: /var/lib/docker/containersname: docker-logsports:- containerPort: 24224resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100mmemory: 512MiterminationGracePeriodSeconds: 10volumes:- name: fluentd-configconfigMap:name: fluentd-config- name: containers-logshostPath:path: /var/logtype: DirectoryOrCreate- name: docker-logshostPath:path: /var/lib/docker/containersrestartPolicy: Always--- apiVersion: v1
kind: Service
metadata:name: fluentdnamespace: defaultlabels:tag: fluentd
spec:type: NodePortports:- nodePort: 30011port: 24224targetPort: 24224protocol: TCPselector:app: fluentdKibana
apiVersion: apps/v1beta1
kind: Deployment
metadata:name: kibananamespace: defaultlabels:tag: kibana
spec:replicas: 1template:metadata:labels:app: kibanaspec:containers:- name: kibanaimage: kibana:6.8.6imagePullPolicy: IfNotPresentenv:- name: discovery.typevalue: single-node- name: XPACK_MONITORING_ENABLEDvalue: trueports:- containerPort: 5601resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100mmemory: 512MiterminationGracePeriodSeconds: 10restartPolicy: Always--- apiVersion: v1
kind: Service
metadata:name: kibananamespace: defaultlabels:tag: kibana
spec:type: NodePortports:- nodePort: 30009port: 5601targetPort: 5601protocol: TCPselector:app: kibana2. Exceptionless日志系统
2.1 Exceptionless核心特性
友好的界面内置日志分类看板按团队和项目管理应用支持Web hook发送异常通知基于.Net Core开源 Exceptionless由UI、Job、Api三个镜像组成。背后依赖了Redis存储和文件存储 文件存储是用来存储应用提交的日志文件然后通过Job将这些文件推送到Elasticsearch中用户可以通过UI和Api从dashboard上查询异常日志 应用程序通过Api推送异常日志 2.2 Exceptionless部署文件
apiVersion: apps/v1beta1
kind: Deployment
metadata:name: exceptionless-apinamespace: defaultlabels:tag: exceptionless-api
spec:replicas: 1template:metadata:labels:app: exceptionless-apispec:containers:- name: exceptionless-apiimage: exceptionless/api:6.0.0imagePullPolicy: IfNotPresentenv:- name: EX_AppModevalue: Production- name: EX_BaseURLvalue: http://localhost:30013- name: EX_ConnectionStrings__Cachevalue: providerredis- name: EX_ConnectionStrings__Elasticsearchvalue: serverhttp://elasticsearch:9200;enable-size-pluginfalse#- name: EX_ConnectionStrings__Email# value: smtps://user:passwordsmtp.host.com:587- name: EX_ConnectionStrings__MessageBusvalue: providerredis- name: EX_ConnectionStrings__Queuevalue: providerredis- name: EX_ConnectionStrings__Redisvalue: serverredis,abortConnectfalse- name: EX_ConnectionStrings__Storagevalue: providerfolder;path/app/storage- name: EX_RunJobsInProcessvalue: falsevolumeMounts:- mountPath: /app/storagename: exceptionless-storageports:- containerPort: 80resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100mmemory: 128MiterminationGracePeriodSeconds: 10volumes:- name: exceptionless-storagehostPath:path: /d/k8s/volumes/exceptionless // 本地存储目录type: DirectoryOrCreaterestartPolicy: Always---apiVersion: v1
kind: Service
metadata:name: exceptionless-apinamespace: defaultlabels:tag: exceptionless-api
spec:type: NodePortports:- nodePort: 30012port: 80targetPort: 80protocol: TCPselector:app: exceptionless-api---apiVersion: apps/v1beta1
kind: Deployment
metadata:name: exceptionless-jobnamespace: defaultlabels:tag: exceptionless-job
spec:replicas: 1template:metadata:labels:app: exceptionless-jobspec:containers:- name: exceptionless-jobimage: exceptionless/job:6.0.0imagePullPolicy: IfNotPresentenv:- name: EX_AppModevalue: Production- name: EX_BaseURLvalue: http://localhost:30013- name: EX_ConnectionStrings__Cachevalue: providerredis- name: EX_ConnectionStrings__Elasticsearchvalue: serverhttp://elasticsearch:9200;enable-size-pluginfalse#- name: EX_ConnectionStrings__Email# value: smtps://user:passwordsmtp.host.com:587- name: EX_ConnectionStrings__MessageBusvalue: providerredis- name: EX_ConnectionStrings__Queuevalue: providerredis- name: EX_ConnectionStrings__Redisvalue: serverredis,abortConnectfalse- name: EX_ConnectionStrings__Storagevalue: providerfolder;path/app/storage- name: EX_RunJobsInProcessvalue: falsevolumeMounts:- mountPath: /app/storagename: exceptionless-job-storageports:- containerPort: 80resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100mmemory: 128MiterminationGracePeriodSeconds: 10volumes:- name: exceptionless-job-storagehostPath:path: /d/k8s/volumes/exceptionlesstype: DirectoryOrCreaterestartPolicy: Always---apiVersion: v1
kind: Service
metadata:name: exceptionless-jobnamespace: defaultlabels:tag: exceptionless-job
spec:type: NodePortports:- nodePort: 30014port: 80targetPort: 80protocol: TCP---
apiVersion: apps/v1beta1
kind: Deployment
metadata:name: exceptionless-uinamespace: defaultlabels:tag: exceptionless-ui
spec:replicas: 1template:metadata:labels:app: exceptionless-uispec:containers:- name: exceptionless-uiimage: exceptionless/ui:2.8.1497imagePullPolicy: IfNotPresentenv:- name: AppModevalue: Production- name: EX_ApiUrlvalue: http://localhost:30012ports:- containerPort: 80resources:limits:cpu: 1000mmemory: 2048Mirequests:cpu: 100mmemory: 128MiterminationGracePeriodSeconds: 10restartPolicy: Always---apiVersion: v1
kind: Service
metadata:name: exceptionless-uinamespace: defaultlabels:tag: exceptionless-ui
spec:type: NodePortports:- nodePort: 30013port: 80targetPort: 80protocol: TCPselector:app: exceptionless-ui// Program
public static int Main(string[] args)
{Log.Logger new LoggerConfiguration().ReadFrom.Configuration(Configuration).MinimumLevel.Debug().Enrich.FromLogContext().WriteTo.Console(new RenderedCompactJsonFormatter())//.WriteTo.Fluentd(localhost, 30011, tag: geektime-ordering-api, restrictedToMinimumLevel: Serilog.Events.LogEventLevel.Debug).CreateLogger();// 将Configuration中的exceptionless读取出来,并绑定到它默认的配置上去Configuration.GetSection(exceptionless).Bind(Exceptionless.ExceptionlessClient.Default.Configuration);try{Log.Information(Starting web host);CreateHostBuilder(args).Build().Run();return 0;}catch (Exception ex){Log.Fatal(ex, Host terminated unexpectedly);return 1;}finally{Log.CloseAndFlush();}
}// 配置文件
exceptionless: {ApiKey: PSaokqmXC8T4xgWal9I0atA8TlYG7Ytz65JvxAL3,ServerUrl: http://localhost:30012},// startup
public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
{app.UseExceptionless();// 放在所有中间件最前面.为了能捕获到所有异常
}
2.3 K8s中使用Exceptionless
配置文件 exceptionless: {ApiKey: PSaokqmXC8T4xgWal9I0atA8TlYG7Ytz65JvxAL3 // 需要是当前项目的秘钥},因为serverUrl地址是共享的地址被配置在env.txt中 要将配置引用进来,需要在deployment.yaml中注册serverUrl为环境变量
通过EFK收集应用的全量日志,Exceptionless收集异常日志
