大型网站建设定制开发,杭州排名推广,河南城乡建设厅网站,四川建设网共享平台目录 一、修改kube-controller和kube-schduler的yaml文件
二、创建service、endpoint、serviceMonitor
三、Prometheus验证
四、创建PrometheusRule资源
五、Prometheus验证 直接上干货
一、修改kube-controller和kube-schduler的yaml文件
注意#xff1a;修改时要一个节…目录 一、修改kube-controller和kube-schduler的yaml文件
二、创建service、endpoint、serviceMonitor
三、Prometheus验证
四、创建PrometheusRule资源
五、Prometheus验证 直接上干货
一、修改kube-controller和kube-schduler的yaml文件
注意修改时要一个节点一个节点的修改等上一个修改的节点服务正常启动后再修改下个节点
kube-controller文件路径/etc/kubernetes/manifests/kube-controller-manager.yaml
kube-scheduler文件路径/etc/kubernetes/manifests/kube-scheduler.yamlvim /etc/kubernetes/manifests/kube-controller-manager.yaml
vim /etc/kubernetes/manifests/kube-scheduler.yaml 二、创建service、endpoint、serviceMonitor
kube-controller-monitor.yaml
apiVersion: v1
kind: Service
metadata:labels:k8s-app: kube-controller-managername: kube-controller-manage-monitornamespace: kube-system
spec:ports:- name: https-metricsport: 10257protocol: TCPtargetPort: 10257sessionAffinity: Nonetype: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:labels:k8s-app: kube-controller-managername: kube-controller-manage-monitornamespace: kube-system
subsets:
- addresses:- ip: 10.50.238.191- ip: 10.50.107.48- ip: 10.50.140.151ports:- name: https-metricsport: 10257protocol: TCP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:labels:k8s-app: kube-controller-managername: kube-controller-managernamespace: kube-system
spec:endpoints:- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/tokeninterval: 30sport: https-metricsscheme: httpstlsConfig:insecureSkipVerify: truejobLabel: k8s-appnamespaceSelector:matchNames:- kube-systemselector:matchLabels:k8s-app: kube-controller-manager
kube-scheduler-monitor.yaml
apiVersion: v1
kind: Service
metadata:labels:k8s-app: kube-schedulername: kube-scheduler-monitornamespace: kube-system
spec:ports:- name: https-metricsport: 10259protocol: TCPtargetPort: 10259sessionAffinity: Nonetype: ClusterIP
---
apiVersion: v1
kind: Endpoints
metadata:labels:k8s-app: kube-schedulername: kube-scheduler-monitornamespace: kube-system
subsets:
- addresses:- ip: 10.50.238.191- ip: 10.50.107.48- ip: 10.50.140.151ports:- name: https-metricsport: 10259protocol: TCP
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:labels:k8s-app: kube-schedulername: kube-schedulernamespace: kube-system
spec:endpoints:- bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/tokeninterval: 30sport: https-metricsscheme: httpstlsConfig:insecureSkipVerify: truejobLabel: k8s-appnamespaceSelector:matchNames:- kube-systemselector:matchLabels:k8s-app: kube-scheduler root10-50-238-191:/home/sunwenbo/prometheus-serviceMonitor/serviceMonitor/kubernetes-cluster# kubectl apply -f ./
service/kube-controller-manage-monitor created
endpoints/kube-controller-manage-monitor created
servicemonitor.monitoring.coreos.com/kube-controller-manager created
service/kube-scheduler-monitor created
endpoints/kube-scheduler-monitor created
servicemonitor.monitoring.coreos.com/kube-scheduler created
三、Prometheus验证 四、创建PrometheusRule资源
kube-controller-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:annotations:meta.helm.sh/release-namespace: cattle-monitoring-systemprometheus-operator-validated: truegeneration: 3labels:app: rancher-monitoringapp.kubernetes.io/instance: rancher-monitoringapp.kubernetes.io/part-of: rancher-monitoringname: kube-controller-managernamespace: cattle-monitoring-system
spec:groups:- name: kube-controller-manager.rulerules:- alert: K8SControllerManagerDownexpr: absent(up{jobkube-controller-manager} 1)for: 1mlabels:severity: criticalcluster: manage-prodannotations:description: There is no running K8S controller manager. Deployments and replication controllers are not making progress.summary: No kubernetes controller manager are reachable- alert: K8SControllerManagerDownexpr: up{jobkube-controller-manager} 0for: 1mlabels:severity: warningcluster: manage-prodannotations:description: kubernetes controller manager {{ $labels.instance }} is down. {{ $labels.instance }} isnt reachablesummary: kubernetes controller manager is down- alert: K8SControllerManagerUserCPUexpr: sum(rate(container_cpu_user_seconds_total{pod~kube-controller-manager.*,container_name!POD}[5m]))by(pod) 5for: 5mlabels:severity: warningcluster: manage-prodannotations:description: kubernetes controller manager {{ $labels.instance }} is user cpu time 5s. {{ $labels.instance }} isnt reachablesummary: kubernetes controller 负载较高超过5s- alert: K8SControllerManagerUseMemoryexpr: sum(rate(container_memory_usage_bytes{pod~kube-controller-manager.*,container_name!POD}[5m])/1024/1024)by(pod) 20for: 5mlabels:severity: infocluster: manage-prodannotations:description: kubernetes controller manager {{ $labels.instance }} is use memory More than 20MBsummary: kubernetes controller 使用内存超过20MB- alert: K8SControllerManagerQueueTimedelayexpr: histogram_quantile(0.99, sum(rate(workqueue_queue_duration_seconds_bucket{jobkubernetes-controller-manager}[5m])) by(le)) 10for: 5mlabels:severity: warningcluster: manage-prodannotations:description: kubernetes controller manager {{ $labels.instance }} is QueueTimedelay More than 10ssummary: kubernetes controller 队列停留时间超过10秒请检查ControllerManager
kube-scheduler-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:annotations:meta.helm.sh/release-namespace: cattle-monitoring-systemprometheus-operator-validated: truegeneration: 3labels:app: rancher-monitoringapp.kubernetes.io/instance: rancher-monitoringapp.kubernetes.io/part-of: rancher-monitoringname: kube-schedulernamespace: cattle-monitoring-system
spec:groups:- name: kube-scheduler.rulerules:- alert: K8SSchedulerDownexpr: absent(up{jobkube-scheduler} 1)for: 1mlabels:severity: criticalcluster: manage-prodannotations:description: There is no running K8S scheduler. New pods are not being assigned to nodes.summary: all k8s scheduler is down- alert: K8SSchedulerDownexpr: up{jobkube-scheduler} 0for: 1mlabels:severity: warningcluster: manage-prodannotations:description: K8S scheduler {{ $labels.instance }} is no running. New pods are not being assigned to nodes.summary: k8s scheduler {{ $labels.instance }} is down- alert: K8SSchedulerUserCPUexpr: sum(rate(container_cpu_user_seconds_total{pod~kube-scheduler.*,container_name!POD}[5m]))by(pod) 1for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: {{$value}}description: kubernetes scheduler {{ $labels.instance }} is user cpu time 1s. {{ $labels.instance }} isnt reachablesummary: kubernetes scheduler 负载较高超过1s,当前值为{{$value}}- alert: K8SSchedulerUseMemoryexpr: sum(rate(container_memory_usage_bytes{pod~kube-scheduler.*,container_name!POD}[5m])/1024/1024)by(pod) 20for: 5mlabels:severity: infocluster: manage-prodannotations:current_value: {{$value}}description: kubernetess scheduler {{ $labels.instance }} is use memory More than 20MBsummary: kubernetes scheduler 使用内存超过20MB,当前值为{{$value}}MB- alert: K8SSchedulerPodPendingexpr: sum(scheduler_pending_pods{jobkubernetes-scheduler})by(queue) 5for: 5mlabels:severity: infocluster: manage-prodannotations:current_value: {{$value}}description: kubernetess scheduler {{ $labels.instance }} is Pending pod More than 5summary: kubernetes scheduler pod无法调度 5,当前值为{{$value}}- alert: K8SSchedulerPodPendingexpr: sum(scheduler_pending_pods{jobkubernetes-scheduler})by(queue) 10for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: {{$value}}description: kubernetess scheduler {{ $labels.instance }} is Pending pod More than 10summary: kubernetes scheduler pod无法调度 10,当前值为{{$value}}- alert: K8SSchedulerPodPendingexpr: sum(rate(scheduler_binding_duration_seconds_count{jobkubernetes-scheduler}[5m])) 1for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: {{$value}}description: kubernetess scheduler {{ $labels.instance }}summary: kubernetes scheduler pod 无法绑定调度有问题当前值为{{$value}}- alert: K8SSchedulerVolumeSpeedexpr: sum(rate(scheduler_volume_scheduling_duration_seconds_count{jobkubernetes-scheduler}[5m])) 1for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: {{$value}}description: kubernetess scheduler {{ $labels.instance }}summary: kubernetes scheduler pod Volume 速度延迟当前值为{{$value}}- alert: K8SSchedulerClientRequestSlowexpr: histogram_quantile(0.99, sum(rate(rest_client_request_duration_seconds_bucket{jobkubernetes-scheduler}[5m])) by (verb, url, le)) 1for: 5mlabels:severity: warningcluster: manage-prodannotations:current_value: {{$value}}description: kubernetess scheduler {{ $labels.instance }}summary: kubernetes scheduler 客户端请求速度延迟当前值为{{$value}}
root10-50-238-191:/home/sunwenbo/prometheus-serviceMonitor/rules# kubectl apply -f kube-controller-rules.yaml
prometheusrule.monitoring.coreos.com/kube-apiserver-rules configured
root10-50-238-191:/home/sunwenbo/prometheus-serviceMonitor/rules# kubectl apply -f kube-scheduler-rules.yaml
prometheusrule.monitoring.coreos.com/kube-apiserver-rules configured
root10-50-238-191:/home/sunwenbo/prometheus-serviceMonitor/rules#
五、Prometheus验证
