{"id":32857,"date":"2025-03-04T11:02:56","date_gmt":"2025-03-04T03:02:56","guid":{"rendered":"https:\/\/www.yunshan.net\/blog\/?p=32857"},"modified":"2025-03-04T11:02:56","modified_gmt":"2025-03-04T03:02:56","slug":"prometheus%e7%9a%84%e6%9e%b6%e6%9e%84%e8%ae%be%e8%ae%a1%e6%9c%89%e5%93%aa%e4%ba%9b%e5%85%b3%e9%94%ae%e7%bb%84%e4%bb%b6%ef%bc%9f","status":"publish","type":"post","link":"https:\/\/www.yunshan.net\/blog\/archives\/32857.html","title":{"rendered":"Prometheus\u7684\u67b6\u6784\u8bbe\u8ba1\u6709\u54ea\u4e9b\u5173\u952e\u7ec4\u4ef6\uff1f"},"content":{"rendered":"<p style=\"text-align: center;\">\n<p><img decoding=\"async\" src=\"https:\/\/www.yunshan.net\/blog\/wp-content\/uploads\/2025\/03\/E6Gm3.jpg\" alt=\"Prometheus\u7684\u67b6\u6784\u8bbe\u8ba1\u6709\u54ea\u4e9b\u5173\u952e\u7ec4\u4ef6\uff1f\" ><\/p>\n<\/p>\n<p>\u5728\u4e91\u539f\u751f\u6280\u672f\u84ec\u52c3\u53d1\u5c55\u7684\u4eca\u5929\uff0c\u76d1\u63a7\u7cfb\u7edf\u5df2\u6210\u4e3a\u4fdd\u969c\u5206\u5e03\u5f0f\u7cfb\u7edf\u7a33\u5b9a\u8fd0\u884c\u7684\u201c\u795e\u7ecf\u7cfb\u7edf\u201d\u3002\u4f5c\u4e3aCNCF\uff08\u4e91\u539f\u751f\u8ba1\u7b97\u57fa\u91d1\u4f1a\uff09\u9996\u4e2a\u6bd5\u4e1a\u7684\u5f00\u6e90\u76d1\u63a7\u9879\u76ee\uff0c<strong><a href=\"https:\/\/www.yunshan.net\/\"   target=\"_blank\" style=\"color:blue\">Prometheus<\/a><\/strong>\u51ed\u501f\u5176\u7075\u6d3b\u7684\u67b6\u6784\u8bbe\u8ba1\u548c\u9ad8\u6548\u7684\u6570\u636e\u5904\u7406\u80fd\u529b\uff0c\u6210\u4e3a\u5bb9\u5668\u5316\u73af\u5883\u76d1\u63a7\u7684\u4e8b\u5b9e\u6807\u51c6\u3002\u5176\u6210\u529f\u80cc\u540e\uff0c<strong>\u6a21\u5757\u5316\u67b6\u6784<\/strong>\u4e0e<strong>\u7ec4\u4ef6\u534f\u540c\u673a\u5236<\/strong>\u7684\u5de7\u5999\u8bbe\u8ba1\u529f\u4e0d\u53ef\u6ca1\u2014\u2014\u8fd9\u79cd\u8bbe\u8ba1\u4e0d\u4ec5\u5b9e\u73b0\u4e86\u6307\u6807\u91c7\u96c6\u3001\u5b58\u50a8\u3001\u544a\u8b66\u7684\u5168\u94fe\u8def\u8986\u76d6\uff0c\u66f4\u901a\u8fc7<strong>\u677e\u8026\u5408\u7ec4\u4ef6<\/strong>\u7684\u6709\u673a\u7ec4\u5408\uff0c\u4e3a\u4e0d\u540c\u89c4\u6a21\u7684\u73af\u5883\u63d0\u4f9b\u4e86\u53ef\u6269\u5c55\u7684\u76d1\u63a7\u89e3\u51b3\u65b9\u6848\u3002\u672c\u6587\u5c06\u6df1\u5ea6\u5256\u6790<a href=\"https:\/\/www.yunshan.net\/\"   target=\"_blank\" style=\"color:blue\">Prometheus<\/a>\u67b6\u6784\u4e2d\u90a3\u4e9b\u652f\u6491\u5176\u9ad8\u6548\u8fd0\u8f6c\u7684\u201c\u9f7f\u8f6e\u201d\u3002<\/p>\n<hr>\n<h3 id=\"\u4e00-\u6838\u5fc3\u5f15\u64ce-prometheus-server\u7684\u4e09\u5927\u6838\u5fc3\u80fd\u529b\">\u4e00\u3001\u6838\u5fc3\u5f15\u64ce\uff1aPrometheus Server\u7684\u4e09\u5927\u6838\u5fc3\u80fd\u529b<\/h3>\n<p>\u4f5c\u4e3a\u6574\u4e2a\u76d1\u63a7\u7cfb\u7edf\u7684\u5927\u8111\uff0c<strong>Prometheus Server<\/strong>\u627f\u62c5\u7740\u6570\u636e\u6293\u53d6\u3001\u5b58\u50a8\u548c\u67e5\u8be2\u7684\u6838\u5fc3\u804c\u8d23\u3002\u5176\u67b6\u6784\u91c7\u7528\u201c\u62c9\u53d6\u6a21\u578b\u201d\uff08Pull-Based\uff09\uff0c\u4e3b\u52a8\u4ece\u914d\u7f6e\u7684\u76ee\u6807\u4e2d\u83b7\u53d6\u6307\u6807\u6570\u636e\uff0c\u8fd9\u79cd\u8bbe\u8ba1\u5929\u7136\u9002\u914d\u52a8\u6001\u53d8\u5316\u7684\u4e91\u73af\u5883\u3002<\/p>\n<ul>\n<li><strong>\u6570\u636e\u6293\u53d6\u5668\uff08Retriever\uff09<\/strong> \u901a\u8fc7HTTP\u534f\u8bae\u5b9a\u671f\u4ece\u914d\u7f6e\u7684<strong>Exporter<\/strong>\u6216<strong>Service Discovery<\/strong>\u76ee\u6807\u62c9\u53d6\u6307\u6807\uff0c\u652f\u6301\u7075\u6d3b\u7684\u6293\u53d6\u95f4\u9694\u914d\u7f6e\uff08\u5982\u6bcf15\u79d2\u6293\u53d6\u4e00\u6b21Kubernetes\u8282\u70b9\u72b6\u6001\uff09\u3002\n<\/li>\n<li><strong>\u65f6\u5e8f\u6570\u636e\u5e93\uff08TSDB\uff09<\/strong> \u91c7\u7528\u81ea\u5b9a\u4e49\u7684\u5b58\u50a8\u683c\u5f0f\uff0c\u4ee5\u5757\uff08Block\uff09\u5f62\u5f0f\u5728\u672c\u5730\u78c1\u76d8\u5b58\u50a8\u6570\u636e\uff0c\u914d\u5408\u5185\u5b58\u4e2d\u7684\u9884\u5199\u65e5\u5fd7\uff08WAL\uff09\u673a\u5236\uff0c\u786e\u4fdd\u6570\u636e\u91c7\u96c6\u4e0d\u4e22\u5931\u3002\u4f8b\u5982\u5728\u8282\u70b9\u5b95\u673a\u6062\u590d\u540e\uff0c\u4ecd\u80fd\u901a\u8fc7WAL\u91cd\u5efa\u6700\u8fd12\u5c0f\u65f6\u7684\u76d1\u63a7\u6570\u636e\u3002\n<\/li>\n<li><strong>\u67e5\u8be2\u5f15\u64ce<\/strong> \u5185\u7f6e\u7684PromQL\u8bed\u8a00\u652f\u6301\u591a\u7ef4\u6570\u636e\u67e5\u8be2\uff0c\u50cf<code>http_requests_total{status=&quot;500&quot;, method=&quot;POST&quot;}<\/code>\u8fd9\u7c7b\u67e5\u8be2\u8bed\u53e5\uff0c\u80fd\u7cbe\u51c6\u5b9a\u4f4d\u7279\u5b9a\u7ef4\u5ea6\u7684\u5f02\u5e38\u6307\u6807\u3002\n<\/li>\n<\/ul>\n<hr>\n<h3 id=\"\u4e8c-\u6570\u636e\u91c7\u96c6\u751f\u6001-exporters\u4e0epushgateway\u7684\u53cc\u91cd\u89e6\u89d2\">\u4e8c\u3001\u6570\u636e\u91c7\u96c6\u751f\u6001\uff1aExporters\u4e0ePushgateway\u7684\u53cc\u91cd\u89e6\u89d2<\/h3>\n<p><strong>Exporters<\/strong>\u4f5c\u4e3a\u76d1\u63a7\u6570\u636e\u7684\u201c\u7ffb\u8bd1\u5b98\u201d\uff0c\u5c06\u5404\u79cd\u7cfb\u7edf\u7684\u539f\u751f\u6570\u636e\u8f6c\u6362\u4e3aPrometheus\u53ef\u8bc6\u522b\u7684\u6307\u6807\u683c\u5f0f\u3002\u5e38\u89c1\u7684Exporter\u5305\u62ec\uff1a<\/p>\n<ul>\n<li><strong>Node Exporter<\/strong>\uff1a\u91c7\u96c6\u4e3b\u673a\u7ea7\u6307\u6807\uff08CPU\/\u5185\u5b58\/\u78c1\u76d8\u4f7f\u7528\u7387\uff09\n<\/li>\n<li><strong>Blackbox Exporter<\/strong>\uff1a\u5b9e\u73b0HTTP\/ICMP\/TCP\u7b49\u7f51\u7edc\u63a2\u6d4b\n<\/li>\n<li><strong>\u81ea\u5b9a\u4e49Exporter<\/strong>\uff1a\u5f00\u53d1\u8005\u53ef\u901a\u8fc7Client Library\uff08\u5982Python\u7684prometheus-client\uff09\u66b4\u9732\u4e1a\u52a1\u6307\u6807\n<\/li>\n<\/ul>\n<p>\u800c<strong>Pushgateway<\/strong>\u5219\u7a81\u7834\u4e86Pull\u6a21\u578b\u7684\u9650\u5236\uff0c\u5141\u8bb8\u77ed\u671f\u4efb\u52a1\uff08\u5982CronJob\uff09\u5728\u9000\u51fa\u524d\u5c06\u6307\u6807\u63a8\u9001\u5230\u7f51\u5173\u3002\u5f53\u5904\u7406\u6279\u5904\u7406\u4f5c\u4e1a\u76d1\u63a7\u65f6\uff0c\u4efb\u52a1\u5b8c\u6210\u540e\u5c06<code>job_duration_seconds<\/code>\u6307\u6807\u63a8\u9001\u5230Pushgateway\uff0c\u518d\u7531Server\u7edf\u4e00\u62c9\u53d6\uff0c\u5b8c\u7f8e\u89e3\u51b3\u4e34\u65f6\u4efb\u52a1\u65e0\u6cd5\u6301\u7eed\u66b4\u9732\u6307\u6807\u7684\u95ee\u9898\u3002<\/p>\n<hr>\n<h3 id=\"\u4e09-\u52a8\u6001\u53d1\u73b0\u673a\u5236-service-discovery\u7684\u667a\u80fd\u9002\u914d\">\u4e09\u3001\u52a8\u6001\u53d1\u73b0\u673a\u5236\uff1aService Discovery\u7684\u667a\u80fd\u9002\u914d<\/h3>\n<p>\u5728\u52a8\u6001\u7f16\u6392\u73af\u5883\u4e0b\uff0c\u4f20\u7edf\u9759\u6001\u914d\u7f6eIP\u7684\u65b9\u5f0f\u663e\u7136\u529b\u4e0d\u4ece\u5fc3\u3002Prometheus\u901a\u8fc7<strong>\u670d\u52a1\u53d1\u73b0\uff08Service Discovery\uff09<\/strong> \u96c6\u6210\u591a\u79cd\u4e91\u5e73\u53f0\uff1a<\/p>\n<ul>\n<li><strong>Kubernetes SD<\/strong>\uff1a\u81ea\u52a8\u53d1\u73b0\u96c6\u7fa4\u4e2d\u7684Pod\/Service\/Endpoint\u53d8\u5316\n<\/li>\n<li><strong>Consul SD<\/strong>\uff1a\u5bf9\u63a5\u670d\u52a1\u7f51\u683c\u4e2d\u7684\u670d\u52a1\u6ce8\u518c\u4fe1\u606f\n<\/li>\n<li><strong>\u6587\u4ef6SD<\/strong>\uff1a\u901a\u8fc7JSON\/YAML\u6587\u4ef6\u52a8\u6001\u52a0\u8f7d\u76ee\u6807\u5217\u8868\n<\/li>\n<\/ul>\n<p>\u914d\u5408<strong>Relabeling<\/strong>\u673a\u5236\uff0c\u53ef\u5b9e\u73b0\u6807\u7b7e\u7684\u52a8\u6001\u6539\u5199\u3002\u4f8b\u5982\u5c06\u4eceKubernetes\u53d1\u73b0\u7684Pod\u5143\u6570\u636e\uff08\u5982<code>__meta_kubernetes_pod_label_app<\/code>\uff09\u8f6c\u6362\u4e3a\u4e1a\u52a1\u6807\u7b7e<code>app=nginx<\/code>\uff0c\u4f7f\u5f97\u76d1\u63a7\u6570\u636e\u5929\u7136\u643a\u5e26\u4e1a\u52a1\u4e0a\u4e0b\u6587\u3002<\/p>\n<hr>\n<h3 id=\"\u56db-\u544a\u8b66\u4e2d\u67a2-alertmanager\u7684\u667a\u80fd\u8def\u7531\u7b56\u7565\">\u56db\u3001\u544a\u8b66\u4e2d\u67a2\uff1aAlertmanager\u7684\u667a\u80fd\u8def\u7531\u7b56\u7565<\/h3>\n<p>\u5f53Prometheus Server\u6839\u636e\u544a\u8b66\u89c4\u5219\uff08\u5982<code>ALERT HighErrorRate IF rate(errors_total[5m]) &gt; 0.1<\/code>\uff09\u89e6\u53d1\u8b66\u62a5\u65f6\uff0c<strong>Alertmanager<\/strong>\u8d1f\u8d23\u53bb\u91cd\u3001\u5206\u7ec4\u548c\u8def\u7531\uff1a<\/p>\n<ul>\n<li><strong>\u6291\u5236\u89c4\u5219\uff08Inhibition\uff09<\/strong>\uff1a\u5f53\u6570\u636e\u5e93\u4e3b\u8282\u70b9\u6545\u969c\u65f6\uff0c\u81ea\u52a8\u6291\u5236\u4ece\u8282\u70b9\u7684\u5197\u4f59\u544a\u8b66\n<\/li>\n<li><strong>\u9759\u9ed8\u914d\u7f6e\uff08Silence\uff09<\/strong>\uff1a\u5728\u7ef4\u62a4\u7a97\u53e3\u671f\u5c4f\u853d\u7279\u5b9a\u544a\u8b66\n<\/li>\n<li><strong>\u591a\u901a\u9053\u901a\u77e5<\/strong>\uff1a\u652f\u6301\u5c06\u4e25\u91cd\u544a\u8b66\u901a\u8fc7Slack\/PagerDuty\u53d1\u9001\uff0c\u800c\u8b66\u544a\u7ea7\u522b\u4fe1\u606f\u901a\u8fc7\u90ae\u4ef6\u901a\u77e5\n<\/li>\n<\/ul>\n<p>\u8fd9\u79cd\u5206\u5c42\u5904\u7406\u673a\u5236\u6709\u6548\u907f\u514d\u4e86\u201c\u544a\u8b66\u98ce\u66b4\u201d\uff0c\u67d0\u7535\u5546\u5e73\u53f0\u66fe\u901a\u8fc7\u4f18\u5316\u5206\u7ec4\u7b56\u7565\uff0c\u5c06\u751f\u4ea7\u73af\u5883\u544a\u8b66\u91cf\u51cf\u5c1170%\u3002<\/p>\n<hr>\n<h3 id=\"\u4e94-\u6570\u636e\u53ef\u89c6\u5316-grafana\u4e0eprometheus\u7684\u9ec4\u91d1\u7ec4\u5408\">\u4e94\u3001\u6570\u636e\u53ef\u89c6\u5316\uff1aGrafana\u4e0ePrometheus\u7684\u9ec4\u91d1\u7ec4\u5408<\/h3>\n<p>\u867d\u7136Prometheus\u81ea\u5e26\u57fa\u7840\u4eea\u8868\u76d8\uff0c\u4f46<strong>Grafana<\/strong>\u51ed\u501f\u4e30\u5bcc\u7684\u53ef\u89c6\u5316\u63d2\u4ef6\u6210\u4e3a\u9996\u9009\u5c55\u793a\u5c42\u3002\u901a\u8fc7\u914d\u7f6ePromQL\u6570\u636e\u6e90\uff0c\u53ef\u6784\u5efa\u5305\u542b\u70ed\u529b\u56fe\uff08Heatmap\uff09\u3001\u72b6\u6001\u56fe\uff08Stat\uff09\u7684\u7efc\u5408\u6027\u76d1\u63a7\u770b\u677f\u3002\u4f8b\u5982\u5728\u5206\u6790API\u6027\u80fd\u65f6\uff0c\u901a\u8fc7<code>histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))<\/code>\u67e5\u8be2P95\u54cd\u5e94\u65f6\u95f4\uff0c\u5e76\u4ee5\u6298\u7ebf\u56fe\u5f62\u5f0f\u5c55\u73b0\u8d8b\u52bf\u53d8\u5316\u3002<\/p>\n<hr>\n<h3 id=\"\u516d-\u53ef\u6269\u5c55\u6027\u8bbe\u8ba1-\u8054\u90a6\u96c6\u7fa4\u4e0e\u8fdc\u7a0b\u5b58\u50a8\">\u516d\u3001\u53ef\u6269\u5c55\u6027\u8bbe\u8ba1\uff1a\u8054\u90a6\u96c6\u7fa4\u4e0e\u8fdc\u7a0b\u5b58\u50a8<\/h3>\n<p>\u4e3a\u5e94\u5bf9\u8d85\u5927\u89c4\u6a21\u76d1\u63a7\u573a\u666f\uff0cPrometheus\u63d0\u4f9b\u4e86<strong>\u8054\u90a6\u96c6\u7fa4\uff08Federation\uff09<\/strong>\u67b6\u6784\u3002\u4e0a\u5c42Prometheus Server\u4ece\u4e0b\u6e38\u5b9e\u4f8b\u805a\u5408\u5173\u952e\u6307\u6807\uff0c\u4f8b\u5982\u5728\u8de8\u56fd\u4e1a\u52a1\u4e2d\uff0c\u533a\u57df\u7ea7Prometheus\u6536\u96c6\u672c\u5730\u6570\u636e\uff0c\u603b\u90e8\u901a\u8fc7<code>federation<\/code>\u63a5\u53e3\u6c47\u805a<code>sum(up{job=&quot;prometheus&quot;}) by (region)<\/code>\u7b49\u5168\u5c40\u72b6\u6001\u6307\u6807\u3002<\/p>\n<p>\u540c\u65f6\u901a\u8fc7<strong>Remote Write\/Read<\/strong>\u63a5\u53e3\uff0c\u53ef\u5c06\u6570\u636e\u957f\u671f\u5b58\u50a8\u5230InfluxDB\u3001TimescaleDB\u7b49\u8fdc\u7aef\u5b58\u50a8\uff0c\u89e3\u51b3\u672c\u5730\u5b58\u50a8\u6269\u5bb9\u96be\u9898\u3002\u67d0\u91d1\u878d\u4f01\u4e1a\u91c7\u7528Thanos\u65b9\u6848\uff0c\u5b9e\u73b0\u4e86PB\u7ea7\u76d1\u63a7\u6570\u636e\u7684\u8de8\u4e2d\u5fc3\u7edf\u4e00\u67e5\u8be2\u3002<\/p>\n<hr>\n<h3 id=\"\u4e03-\u5b89\u5168\u4e0e\u9ad8\u53ef\u7528\u5b9e\u8df5\">\u4e03\u3001\u5b89\u5168\u4e0e\u9ad8\u53ef\u7528\u5b9e\u8df5<\/h3>\n<p>\u5728\u751f\u4ea7\u90e8\u7f72\u4e2d\uff0c\u901a\u5e38\u4f1a\u901a\u8fc7\u4ee5\u4e0b\u65b9\u5f0f\u589e\u5f3a\u7a33\u5b9a\u6027\uff1a<\/p>\n<ul>\n<li><strong>\u53cc\u6d3b\u90e8\u7f72<\/strong>\uff1a\u8fd0\u884c\u4e24\u4e2a\u5b8c\u5168\u76f8\u540c\u7684Prometheus Server\u5b9e\u4f8b\uff0c\u5171\u4eab\u76f8\u540c\u7684\u914d\u7f6e\u548c\u5b58\u50a8\u5377\n<\/li>\n<li><strong>TLS\u52a0\u5bc6<\/strong>\uff1a\u4e3aExporter\u914d\u7f6eHTTPS\u7aef\u70b9\u53ca\u53cc\u5411\u8bc1\u4e66\u8ba4\u8bc1\n<\/li>\n<li><strong>\u8d44\u6e90\u9694\u79bb<\/strong>\uff1a\u4f7f\u7528cgroups\u9650\u5236\u5185\u5b58\u7528\u91cf\uff0c\u907f\u514dOOM\uff08\u5185\u5b58\u6ea2\u51fa\uff09\u5bfc\u81f4\u8fdb\u7a0b\u5d29\u6e83\n<\/li>\n<\/ul>\n<p>\u901a\u8fc7\u8fd9\u4e03\u5927\u7ec4\u4ef6\u7684\u7cbe\u5bc6\u534f\u4f5c\uff0cPrometheus\u6784\u5efa\u4e86\u4e00\u4e2a\u65e2\u80fd\u5728\u5355\u8282\u70b9\u5feb\u901f\u90e8\u7f72\uff0c\u53c8\u80fd\u652f\u6491\u6570\u5343\u8282\u70b9\u96c6\u7fa4\u7684\u5f39\u6027\u76d1\u63a7\u4f53\u7cfb\u3002\u6b63\u5982CNCF\u6280\u672f\u76d1\u7763\u59d4\u5458\u4f1a\u6240\u8a00\uff1a\u201c\u5b83\u7684\u7ec4\u4ef6\u5316\u67b6\u6784\u5b8c\u7f8e\u8be0\u91ca\u4e86Unix\u54f2\u5b66\u2014\u2014\u6bcf\u4e2a\u7ec4\u4ef6\u53ea\u505a\u597d\u4e00\u4ef6\u4e8b\uff0c\u5e76\u901a\u8fc7\u6e05\u6670\u5b9a\u4e49\u7684\u63a5\u53e3\u534f\u540c\u5de5\u4f5c\u3002\u201d\u8fd9\u79cd\u8bbe\u8ba1\u54f2\u5b66\uff0c\u6b63\u662fPrometheus\u5728\u4e91\u539f\u751f\u76d1\u63a7\u9886\u57df\u6301\u7eed\u9886\u8dd1\u7684\u6838\u5fc3\u5bc6\u7801\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\u5728\u4e91\u539f\u751f\u6280\u672f\u84ec\u52c3\u53d1\u5c55\u7684\u4eca\u5929\uff0c\u76d1\u63a7\u7cfb\u7edf\u5df2\u6210\u4e3a\u4fdd\u969c\u5206\u5e03\u5f0f\u7cfb\u7edf\u7a33\u5b9a\u8fd0\u884c\u7684\u201c\u795e\u7ecf\u7cfb\u7edf\u201d\u3002\u4f5c\u4e3aCNCF\uff08\u4e91\u539f\u751f\u8ba1\u7b97\u57fa\u91d1\u4f1a\uff09\u9996\u4e2a\u6bd5\u4e1a\u7684\u5f00\u6e90\u76d1\u63a7\u9879\u76ee\uff0cPrometheus\u51ed\u501f\u5176\u7075\u6d3b\u7684\u67b6\u6784\u8bbe\u8ba1\u548c\u9ad8\u6548\u7684\u6570\u636e\u5904\u7406\u80fd\u529b\uff0c\u6210\u4e3a\u5bb9\u5668\u5316\u73af\u5883\u76d1\u63a7\u7684\u4e8b\u5b9e\u6807\u51c6\u3002\u5176\u6210\u529f\u80cc\u540e\uff0c\u6a21\u5757\u5316\u67b6\u6784\u4e0e\u7ec4\u4ef6\u534f\u540c\u673a\u5236\u7684\u5de7\u5999\u8bbe\u8ba1\u529f\u4e0d\u53ef\u6ca1\u2014\u2014\u8fd9\u79cd\u8bbe\u8ba1\u4e0d\u4ec5\u5b9e\u73b0\u4e86\u6307\u6807\u91c7\u96c6\u3001\u5b58\u50a8\u3001\u544a\u8b66\u7684\u5168\u94fe\u8def\u8986\u76d6\uff0c\u66f4\u901a\u8fc7\u677e\u8026\u5408\u7ec4\u4ef6\u7684\u6709\u673a\u7ec4\u5408\uff0c\u4e3a\u4e0d\u540c\u89c4\u6a21\u7684\u73af\u5883\u63d0\u4f9b\u4e86\u53ef\u6269\u5c55\u7684\u76d1\u63a7\u89e3\u51b3\u65b9\u6848\u3002\u672c\u6587\u5c06\u6df1\u5ea6\u5256\u6790Prometheus\u67b6\u6784\u4e2d\u90a3\u4e9b\u652f\u6491\u5176\u9ad8\u6548\u8fd0\u8f6c\u7684\u201c\u9f7f\u8f6e\u201d\u3002 \u4e00\u3001\u6838\u5fc3\u5f15\u64ce\uff1aPrometheus Server\u7684\u4e09\u5927\u6838\u5fc3\u80fd\u529b \u4f5c\u4e3a\u6574\u4e2a\u76d1\u63a7\u7cfb\u7edf\u7684\u5927\u8111\uff0cPrometheus Server\u627f\u62c5\u7740\u6570\u636e\u6293\u53d6\u3001\u5b58\u50a8\u548c\u67e5\u8be2\u7684\u6838\u5fc3\u804c\u8d23\u3002\u5176\u67b6\u6784\u91c7\u7528\u201c\u62c9\u53d6\u6a21\u578b\u201d\uff08Pull-Based\uff09\uff0c\u4e3b\u52a8\u4ece\u914d\u7f6e\u7684\u76ee\u6807\u4e2d\u83b7\u53d6\u6307\u6807\u6570\u636e\uff0c\u8fd9\u79cd\u8bbe\u8ba1\u5929\u7136\u9002\u914d\u52a8\u6001\u53d8\u5316\u7684\u4e91\u73af\u5883\u3002 \u6570\u636e\u6293\u53d6\u5668\uff08Retriever\uff09 \u901a\u8fc7HTTP\u534f\u8bae\u5b9a\u671f\u4ece\u914d\u7f6e\u7684Exporter\u6216Service Discovery\u76ee\u6807\u62c9\u53d6\u6307\u6807\uff0c\u652f\u6301\u7075\u6d3b\u7684\u6293\u53d6\u95f4\u9694\u914d\u7f6e\uff08\u5982\u6bcf15\u79d2\u6293\u53d6\u4e00\u6b21Kubernetes\u8282\u70b9\u72b6\u6001\uff09\u3002 \u65f6\u5e8f\u6570\u636e\u5e93\uff08TSDB\uff09 \u91c7\u7528\u81ea\u5b9a\u4e49\u7684\u5b58\u50a8\u683c\u5f0f\uff0c\u4ee5\u5757\uff08Block\uff09\u5f62\u5f0f\u5728\u672c\u5730\u78c1\u76d8\u5b58\u50a8\u6570\u636e\uff0c\u914d\u5408\u5185\u5b58\u4e2d\u7684\u9884\u5199\u65e5\u5fd7\uff08WAL\uff09\u673a\u5236\uff0c\u786e\u4fdd\u6570\u636e\u91c7\u96c6\u4e0d\u4e22\u5931\u3002\u4f8b\u5982\u5728\u8282\u70b9\u5b95\u673a\u6062\u590d\u540e\uff0c\u4ecd\u80fd\u901a\u8fc7WAL\u91cd\u5efa\u6700\u8fd12\u5c0f\u65f6\u7684\u76d1\u63a7\u6570\u636e\u3002 \u67e5\u8be2\u5f15\u64ce \u5185\u7f6e\u7684PromQL\u8bed\u8a00\u652f\u6301\u591a\u7ef4\u6570\u636e\u67e5\u8be2\uff0c\u50cfhttp_requests_total{status=&quot;500&quot;, method=&quot;POST&quot;}\u8fd9\u7c7b\u67e5\u8be2\u8bed\u53e5\uff0c\u80fd\u7cbe\u51c6\u5b9a\u4f4d\u7279\u5b9a\u7ef4\u5ea6\u7684\u5f02\u5e38\u6307\u6807\u3002 \u4e8c\u3001\u6570\u636e\u91c7\u96c6\u751f\u6001\uff1aExporters\u4e0ePushgateway\u7684\u53cc\u91cd\u89e6\u89d2 Exporters\u4f5c\u4e3a\u76d1\u63a7\u6570\u636e\u7684\u201c\u7ffb\u8bd1\u5b98\u201d\uff0c\u5c06\u5404\u79cd\u7cfb\u7edf\u7684\u539f\u751f\u6570\u636e\u8f6c\u6362\u4e3aPrometheus\u53ef\u8bc6\u522b\u7684\u6307\u6807\u683c\u5f0f\u3002\u5e38\u89c1\u7684Exporter\u5305\u62ec\uff1a Node Exporter\uff1a\u91c7\u96c6\u4e3b\u673a\u7ea7\u6307\u6807\uff08CPU\/\u5185\u5b58\/\u78c1\u76d8\u4f7f\u7528\u7387\uff09 Blackbox Exporter\uff1a\u5b9e\u73b0HTTP\/ICMP\/TCP\u7b49\u7f51\u7edc\u63a2\u6d4b \u81ea\u5b9a\u4e49Exporter\uff1a\u5f00\u53d1\u8005\u53ef\u901a\u8fc7Client Library\uff08\u5982Python\u7684prometheus-client\uff09\u66b4\u9732\u4e1a\u52a1\u6307\u6807 \u800cPushgateway\u5219\u7a81\u7834\u4e86Pull\u6a21\u578b\u7684\u9650\u5236\uff0c\u5141\u8bb8\u77ed\u671f\u4efb\u52a1\uff08\u5982CronJob\uff09\u5728\u9000\u51fa\u524d\u5c06\u6307\u6807\u63a8\u9001\u5230\u7f51\u5173\u3002\u5f53\u5904\u7406\u6279\u5904\u7406\u4f5c\u4e1a\u76d1\u63a7\u65f6\uff0c\u4efb\u52a1\u5b8c\u6210\u540e\u5c06job_duration_seconds\u6307\u6807\u63a8\u9001\u5230Pushgateway\uff0c\u518d\u7531Server\u7edf\u4e00\u62c9\u53d6\uff0c\u5b8c\u7f8e\u89e3\u51b3\u4e34\u65f6\u4efb\u52a1\u65e0\u6cd5\u6301\u7eed\u66b4\u9732\u6307\u6807\u7684\u95ee\u9898\u3002 \u4e09\u3001\u52a8\u6001\u53d1\u73b0\u673a\u5236\uff1aService Discovery\u7684\u667a\u80fd\u9002\u914d \u5728\u52a8\u6001\u7f16\u6392\u73af\u5883\u4e0b\uff0c\u4f20\u7edf\u9759\u6001\u914d\u7f6eIP\u7684\u65b9\u5f0f\u663e\u7136\u529b\u4e0d\u4ece\u5fc3\u3002Prometheus\u901a\u8fc7\u670d\u52a1\u53d1\u73b0\uff08Service Discovery\uff09 \u96c6\u6210\u591a\u79cd\u4e91\u5e73\u53f0\uff1a Kubernetes SD\uff1a\u81ea\u52a8\u53d1\u73b0\u96c6\u7fa4\u4e2d\u7684Pod\/Service\/Endpoint\u53d8\u5316 Consul SD\uff1a\u5bf9\u63a5\u670d\u52a1\u7f51\u683c\u4e2d\u7684\u670d\u52a1\u6ce8\u518c\u4fe1\u606f \u6587\u4ef6SD\uff1a\u901a\u8fc7JSON\/YAML\u6587\u4ef6\u52a8\u6001\u52a0\u8f7d\u76ee\u6807\u5217\u8868 \u914d\u5408Relabeling\u673a\u5236\uff0c\u53ef\u5b9e\u73b0\u6807\u7b7e\u7684\u52a8\u6001\u6539\u5199\u3002\u4f8b\u5982\u5c06\u4eceKubernetes\u53d1\u73b0\u7684Pod\u5143\u6570\u636e\uff08\u5982__meta_kubernetes_pod_label_app\uff09\u8f6c\u6362\u4e3a\u4e1a\u52a1\u6807\u7b7eapp=nginx\uff0c\u4f7f\u5f97\u76d1\u63a7\u6570\u636e\u5929\u7136\u643a\u5e26\u4e1a\u52a1\u4e0a\u4e0b\u6587\u3002 \u56db\u3001\u544a\u8b66\u4e2d\u67a2\uff1aAlertmanager\u7684\u667a\u80fd\u8def\u7531\u7b56\u7565 \u5f53Prometheus Server\u6839\u636e\u544a\u8b66\u89c4\u5219\uff08\u5982ALERT HighErrorRate IF rate(errors_total[5m]) &gt; 0.1\uff09\u89e6\u53d1\u8b66\u62a5\u65f6\uff0cAlertmanager\u8d1f\u8d23\u53bb\u91cd\u3001\u5206\u7ec4\u548c\u8def\u7531\uff1a \u6291\u5236\u89c4\u5219\uff08Inhibition\uff09\uff1a\u5f53\u6570\u636e\u5e93\u4e3b\u8282\u70b9\u6545\u969c\u65f6\uff0c\u81ea\u52a8\u6291\u5236\u4ece\u8282\u70b9\u7684\u5197\u4f59\u544a\u8b66 \u9759\u9ed8\u914d\u7f6e\uff08Silence\uff09\uff1a\u5728\u7ef4\u62a4\u7a97\u53e3\u671f\u5c4f\u853d\u7279\u5b9a\u544a\u8b66 \u591a\u901a\u9053\u901a\u77e5\uff1a\u652f\u6301\u5c06\u4e25\u91cd\u544a\u8b66\u901a\u8fc7Slack\/PagerDuty\u53d1\u9001\uff0c\u800c\u8b66\u544a\u7ea7\u522b\u4fe1\u606f\u901a\u8fc7\u90ae\u4ef6\u901a\u77e5 \u8fd9\u79cd\u5206\u5c42\u5904\u7406\u673a\u5236\u6709\u6548\u907f\u514d\u4e86\u201c\u544a\u8b66\u98ce\u66b4\u201d\uff0c\u67d0\u7535\u5546\u5e73\u53f0\u66fe\u901a\u8fc7\u4f18\u5316\u5206\u7ec4\u7b56\u7565\uff0c\u5c06\u751f\u4ea7\u73af\u5883\u544a\u8b66\u91cf\u51cf\u5c1170%\u3002 \u4e94\u3001\u6570\u636e\u53ef\u89c6\u5316\uff1aGrafana\u4e0ePrometheus\u7684\u9ec4\u91d1\u7ec4\u5408 \u867d\u7136Prometheus\u81ea\u5e26\u57fa\u7840\u4eea\u8868\u76d8\uff0c\u4f46Grafana\u51ed\u501f\u4e30\u5bcc\u7684\u53ef\u89c6\u5316\u63d2\u4ef6\u6210\u4e3a\u9996\u9009\u5c55\u793a\u5c42\u3002\u901a\u8fc7\u914d\u7f6ePromQL\u6570\u636e\u6e90\uff0c\u53ef\u6784\u5efa\u5305\u542b\u70ed\u529b\u56fe\uff08Heatmap\uff09\u3001\u72b6\u6001\u56fe\uff08Stat\uff09\u7684\u7efc\u5408\u6027\u76d1\u63a7\u770b\u677f\u3002\u4f8b\u5982\u5728\u5206\u6790API\u6027\u80fd\u65f6\uff0c\u901a\u8fc7histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))\u67e5\u8be2P95\u54cd\u5e94\u65f6\u95f4\uff0c\u5e76\u4ee5\u6298\u7ebf\u56fe\u5f62\u5f0f\u5c55\u73b0\u8d8b\u52bf\u53d8\u5316\u3002 \u516d\u3001\u53ef\u6269\u5c55\u6027\u8bbe\u8ba1\uff1a\u8054\u90a6\u96c6\u7fa4\u4e0e\u8fdc\u7a0b\u5b58\u50a8 \u4e3a\u5e94\u5bf9\u8d85\u5927\u89c4\u6a21\u76d1\u63a7\u573a\u666f\uff0cPrometheus\u63d0\u4f9b\u4e86\u8054\u90a6\u96c6\u7fa4\uff08Federation\uff09\u67b6\u6784\u3002\u4e0a\u5c42Prometheus Server\u4ece\u4e0b\u6e38\u5b9e\u4f8b\u805a\u5408\u5173\u952e\u6307\u6807\uff0c\u4f8b\u5982\u5728\u8de8\u56fd\u4e1a\u52a1\u4e2d\uff0c\u533a\u57df\u7ea7Prometheus\u6536\u96c6\u672c\u5730\u6570\u636e\uff0c\u603b\u90e8\u901a\u8fc7federation\u63a5\u53e3\u6c47\u805asum(up{job=&quot;prometheus&quot;}) by (region)\u7b49\u5168\u5c40\u72b6\u6001\u6307\u6807\u3002 \u540c\u65f6\u901a\u8fc7Remote Write\/Read\u63a5\u53e3\uff0c\u53ef\u5c06\u6570\u636e\u957f\u671f\u5b58\u50a8\u5230InfluxDB\u3001TimescaleDB\u7b49\u8fdc\u7aef\u5b58\u50a8\uff0c\u89e3\u51b3\u672c\u5730\u5b58\u50a8\u6269\u5bb9\u96be\u9898\u3002\u67d0\u91d1\u878d\u4f01\u4e1a\u91c7\u7528Thanos\u65b9\u6848\uff0c\u5b9e\u73b0\u4e86PB\u7ea7\u76d1\u63a7\u6570\u636e\u7684\u8de8\u4e2d\u5fc3\u7edf\u4e00\u67e5\u8be2\u3002 [&hellip;]<\/p>\n","protected":false},"author":4,"featured_media":32858,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[14],"tags":[],"class_list":["post-32857","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-product-news"],"_links":{"self":[{"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/posts\/32857","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/comments?post=32857"}],"version-history":[{"count":1,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/posts\/32857\/revisions"}],"predecessor-version":[{"id":32859,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/posts\/32857\/revisions\/32859"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/media\/32858"}],"wp:attachment":[{"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/media?parent=32857"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/categories?post=32857"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/tags?post=32857"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}