Kubernetes系列之监控Metres-server实战篇


本次系列使用的所需部署包版本都使用的目前最新的或最新稳定版,安装包地址请到公众号内回复【K8s实战】获取

介绍

从 Kubernetes 1.8 开始,Kubernetes 通过 Metrics API 获取资源使用指标,例如容器 CPU 和内存使用情况。这些度量指标可以由用户直接访问,例如通过使用kubectl top 命令,或者使用集群中的控制器。

Metrics API: 通过 Metrics API,您可以获得 node 或 pod 当前的资源使用情况(但是不存储)。

大致是说它符合 kubernetes 的监控架构设计,受 heapster 项目启发,并且比 heapster 优势在于:

访问不需要 apiserver 的代理机制,提供认证和授权等;

很多集群内组件依赖它(HPA,scheduler,kubectl top),因此它应该在集群中默认运行;

下载编排

[root@master-01 opt]# git clone https://github.com/kubernetes-incubator/metrics-server[root@master-01 opt]# cd metrics-server/deploy/1.8+/

创建metrics-server证书

创建签名请求

[root@master-01 1.8+]# cd /etc/kubernetes/ssl/[root@master-01 ssl]# cat > metrics-server-csr.json <<EOF{\"CN\": \"aggregator\",\"hosts\": [],\"key\": {\"algo\": \"rsa\",\"size\": 2048},\"names\": [{\"C\": \"CN\",\"ST\": \"Hangzhou\",\"L\": \"Hangzhou\",\"O\": \"k8s\",\"OU\": \"4Paradigm\"}]}EOF

创建证书和私钥

[root@master-01 ssl]# cfssl gencert -ca=/etc/kubernetes/ssl/ca.pem -ca-key=/etc/kubernetes/ssl/ca-key.pem -config=/etc/kubernetes/ssl/ca-config.json -profile=kubernetes metrics-server-csr.json|cfssljson -bare metrics-server2019/03/13 15:23:01 [INFO] generate received request2019/03/13 15:23:01 [INFO] received CSR2019/03/13 15:23:01 [INFO] generating key: rsa-20482019/03/13 15:23:01 [INFO] encoded CSR2019/03/13 15:23:01 [INFO] signed certificate with serial number 1026675139058810263099374133507485748972230132012019/03/13 15:23:01 [WARNING] This certificate lacks a \"hosts\" field. This makes it unsuitable forwebsites. For more information see the Baseline Requirements for the Issuance and Managementof Publicly-Trusted Certificates, v.1.1.6, from the CA/Browser Forum (https://cabforum.org);specifically, section 10.2.3 (\"Information Requirements\").

同步证书

同步证书到master-2、master-03

[root@master-01 ssl]# scp metrics-server-key.pem metrics-server.pem 192.168.209.131:/etc/kubernetes/ssl/[root@master-01 ssl]# scp metrics-server-key.pem metrics-server.pem 192.168.209.132:/etc/kubernetes/ssl/

开启聚合配置

修改kube-apiserver配置文件来支持metres-server,加入如下启动参数来启用aggregation layer:

--proxy-client-cert-file=/etc/kubernetes/ssl/metrics-server.pem \\--proxy-client-key-file=/etc/kubernetes/ssl/metrics-server-key.pem \\--runtime-config=api/all=true \\--requestheader-client-ca-file=/etc/kubernetes/ssl/ca.pem \\--requestheader-allowed-names=aggregator \\--requestheader-extra-headers-prefix=X-Remote-Extra- \\--requestheader-group-headers=X-Remote-Group \\--requestheader-username-headers=X-Remote-User

--requestheader-XXX、--proxy-client-XXX 是 kube-apiserver 的 aggregator layer 相关的配置参数,metrics-server & HPA 需要使用。

--requestheader-client-ca-file:用于签名 --proxy-client-cert-file 和 --proxy-client-key-file 指定的证书;在启用了 metric aggregator 时使用。

如果 --requestheader-allowed-names 不为空,则--proxy-client-cert-file 证书的 CN 必须位于 allowed-names 中,默认为 aggregator。

注意:需要重启三台主控的kube-apiserver

如果不开启聚合配置可能会报如下错误

这是因为没用开启聚合层

I0313 05:18:36.447202 1 serving.go:273] Generated self-signed cert (apiserver.local.config/certificates/apiserver.crt, apiserver.local.config/certificates/apiserver.key)Error: cluster doesn\'t provide requestheader-client-ca-file

修改编排文件

在metrics-server-deployment.yaml文件中containers字段下添加如下

command:- /metrics-server- --kubelet-insecure-tls- --kubelet-preferred-address-types=InternalIP

如果不添加 稍后可能会报如下错误

E0313 08:23:41.193222 1 manager.go:102] unable to fully collect metrics: [unable to fully scrape metrics from source kubelet_summary:192.168.209.130: unable to fetch metrics from Kubelet 192.168.209.130 (192.168.209.130): Get https://192.168.209.130:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:192.168.209.131: unable to fetch metrics from Kubelet 192.168.209.131 (192.168.209.131): Get https://192.168.209.131:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:192.168.209.132: unable to fetch metrics from Kubelet 192.168.209.132 (192.168.209.132): Get https://192.168.209.132:10250/stats/summary/: x509: certificate signed by unknown authority, unable to fully scrape metrics from source kubelet_summary:192.168.209.133: unable to fetch metrics from Kubelet 192.168.209.133 (192.168.209.133): Get https://192.168.209.133:10250/stats/summary/: x509: certificate signed by unknown authority]

资料

https://github.com/kubernetes-incubator/metrics-server/issues/67

https://github.com/mattkelly/metrics-server/commit/bfddc174c783290cb86d6da2fe1182d53a3b9bd5

gcr.io的镜像访问不到的话需要将metrics-server-deployment.yaml中的镜像替换为:registry.cn-beijing.aliyuncs.com/minminmsn/metrics-server:v0.3.1

创建metrics-server

[root@master-01 1.8+]# kubectl apply -f ./

查看服务状态

[root@master-01 1.8+]# kubectl get pod -nkube-systemNAME READY STATUS RESTARTS AGEmetrics-server-7c499cd69d-499js 1/1 Running 0 14s

测试功能

可以看到资源使用信息采集到了

[root@master-01 1.8+]# kubectl top pods --all-namespacesNAMESPACE NAME CPU(cores) MEMORY(bytes) default dnstools-6b77cc4988-b5smz 0m 2Mi default nginx-7899755b7-rgdch 0m 2Mi default tests-mychart-7d84ff968f-76d2l 1m 3Mi default wordpress-test-mariadb-59cfd7c475-27chl 5m 116Mi default wordpress-test-wordpress-6fc9b7cc7f-b2nfq 4m 149Mi ingress-nginx grafana-69549786b6-d78nv 1m 30Mi ingress-nginx prometheus-server-8658d8cdbb-4qps2 1m 20Mi kube-system coredns-5d668bd598-4xxwn 3m 13Mi kube-system coredns-5d668bd598-f5g96 2m 9Mi kube-system kubernetes-dashboard-cb55bd5bd-gc84g 1m 19Mi kube-system metrics-server-84f9775b88-gh7x7 2m 16Mi kube-system tiller-deploy-87d7c6dfb-kxj7p 1m 9Mi monitoring kube-state-metrics-6f8967c6c5-nzkxp 2m 30Mi monitoring node-exporter-4n9wj 1m 8Mi monitoring node-exporter-5wtgw 0m 8Mi monitoring node-exporter-gdj8f 1m 11Mi monitoring node-exporter-p96zj 1m 9Mi monitoring prometheus-operator-795895d784-v569s 1m 10Mi [root@master-01 1.8+]# kubectl top nodesNAME CPU(cores) CPU% MEMORY(bytes) MEMORY% 192.168.209.130 158m 7% 1965Mi 53% 192.168.209.131 103m 5% 1859Mi 50% 192.168.209.132 123m 6% 2152Mi 58% 192.168.209.133 38m 1% 1022Mi 27%

通过 kube-apiserver接口访问

https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/nodes 

https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/nodes/ 

https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/pods 

https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/namespace/pods/

[root@master-01 1.8+]# curl -k https://192.168.209.130:6443/apis/metrics.k8s.io/v1beta1/pods {\"kind\": \"PodMetricsList\",\"apiVersion\": \"metrics.k8s.io/v1beta1\",\"metadata\": {\"selfLink\": \"/apis/metrics.k8s.io/v1beta1/pods\"},\"items\": [{\"metadata\": {\"name\": \"coredns-5d668bd598-f5g96\",\"namespace\": \"kube-system\",\"selfLink\": \"/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/coredns-5d668bd598-f5g96\",\"creationTimestamp\": \"2019-03-13T09:23:58Z\"},\"timestamp\": \"2019-03-13T09:23:17Z\",\"window\": \"30s\",\"containers\": [{\"name\": \"coredns\",\"usage\": {\"cpu\": \"2073503n\",\"memory\": \"9628Ki\"}...............

浏览器访问

这里的30000端口是我把metrics-server端口改成NodePort了。

好了,进行到这,metrics就部署完了,敬请期待后续分享,谢谢!

END

如果您觉得不错,请别忘了转发、分享、点赞让更多的人去学习, 您的举手之劳,就是对小编最好的支持,非常感谢!

人已赞赏
随笔日记

git 入门教程之撤销更改

2020-11-9 3:49:03

随笔日记

GIT初始学习记录

2020-11-9 3:49:05

0 条回复 A文章作者 M管理员
    暂无讨论,说说你的看法吧
个人中心
购物车
优惠劵
今日签到
有新私信 私信列表
搜索