{"id":35431,"date":"2025-03-19T09:52:32","date_gmt":"2025-03-19T01:52:32","guid":{"rendered":"https:\/\/www.yunshan.net\/blog\/?p=35431"},"modified":"2025-03-19T09:52:32","modified_gmt":"2025-03-19T01:52:32","slug":"prometheus%e7%9a%84%e6%80%a7%e8%83%bd%e4%bc%98%e5%8c%96%e6%9c%89%e5%93%aa%e4%ba%9b%e7%ad%96%e7%95%a5%ef%bc%9f","status":"publish","type":"post","link":"https:\/\/www.yunshan.net\/blog\/archives\/35431.html","title":{"rendered":"Prometheus\u7684\u6027\u80fd\u4f18\u5316\u6709\u54ea\u4e9b\u7b56\u7565\uff1f"},"content":{"rendered":"

\n

\"Prometheus\u7684\u6027\u80fd\u4f18\u5316\u6709\u54ea\u4e9b\u7b56\u7565\uff1f\"<\/p>\n<\/p>\n

\u968f\u7740\u4e91\u539f\u751f\u6280\u672f\u7684\u5feb\u901f\u53d1\u5c55\uff0cPrometheus<\/a>\u5df2\u6210\u4e3a\u76d1\u63a7\u9886\u57df\u7684\u6838\u5fc3\u5de5\u5177\u4e4b\u4e00\u3002\u7136\u800c\uff0c\u968f\u7740\u6570\u636e\u91cf\u7684\u6307\u6570\u7ea7\u589e\u957f\u548c\u5206\u5e03\u5f0f\u7cfb\u7edf\u7684\u590d\u6742\u6027\u63d0\u5347\uff0c\u8bb8\u591a\u7528\u6237\u53d1\u73b0\u9ed8\u8ba4\u914d\u7f6e\u4e0b\u7684Prometheus<\/a>\u53ef\u80fd\u9762\u4e34\u67e5\u8be2\u5ef6\u8fdf\u9ad8\u3001\u5b58\u50a8\u538b\u529b\u5927\u7b49\u95ee\u9898\u3002\u5982\u4f55\u901a\u8fc7\u79d1\u5b66\u4f18\u5316\u7b56\u7565\u91ca\u653ePrometheus\u7684\u6f5c\u5728\u6027\u80fd\uff1f<\/strong> \u672c\u6587\u5c06\u4ece\u5b58\u50a8\u673a\u5236\u3001\u67e5\u8be2\u6548\u7387\u3001\u914d\u7f6e\u8c03\u4f18\u7b49\u591a\u4e2a\u89d2\u5ea6\uff0c\u6df1\u5165\u63a2\u8ba8\u63d0\u5347Prometheus\u6027\u80fd\u7684\u5b9e\u8df5\u65b9\u6cd5\u3002<\/p>\n

\n

\u4e00\u3001\u4f18\u5316\u5b58\u50a8\u673a\u5236\uff1a\u4ece\u6839\u6e90\u964d\u4f4e\u8d44\u6e90\u6d88\u8017<\/strong><\/h3>\n

Prometheus\u7684TSDB\uff08\u65f6\u95f4\u5e8f\u5217\u6570\u636e\u5e93\uff09\u662f\u5176\u6027\u80fd\u8868\u73b0\u7684\u6838\u5fc3\u3002\u901a\u8fc7\u8c03\u6574\u5b58\u50a8\u7b56\u7565\uff0c\u53ef\u663e\u8457\u51cf\u5c11\u78c1\u76d8\u5360\u7528\u548c\u5185\u5b58\u538b\u529b<\/strong>\uff1a<\/p>\n

    \n
  1. \u542f\u7528\u5757\u538b\u7f29\u4e0e\u5408\u5e76<\/strong>
    \nTSDB\u9ed8\u8ba4\u6bcf2\u5c0f\u65f6\u5c06\u5185\u5b58\u4e2d\u7684\u6570\u636e\u5199\u5165\u78c1\u76d8\u5757\uff08block\uff09\u3002\u901a\u8fc7\u8c03\u6574--storage.tsdb.min-block-duration<\/code>\u548c--storage.tsdb.max-block-duration<\/code>\u53c2\u6570\uff0c\u53ef\u4ee5\u5e73\u8861\u6570\u636e\u5199\u5165\u9891\u7387\u4e0e\u67e5\u8be2\u6548\u7387\u3002\u4f8b\u5982\uff0c\u5728\u6570\u636e\u91cf\u6fc0\u589e\u7684\u73af\u5883\u4e2d\u5c06\u5757\u5468\u671f\u7f29\u77ed\u81f31\u5c0f\u65f6\uff0c\u53ef\u907f\u514d\u56e0\u5355\u5757\u8fc7\u5927\u5bfc\u81f4\u67e5\u8be2\u5ef6\u8fdf\u3002\n<\/li>\n
  2. \u7cbe\u7ec6\u5316\u4fdd\u7559\u7b56\u7565<\/strong>
    \n\u4f7f\u7528--storage.tsdb.retention.time<\/code>\u63a7\u5236\u6570\u636e\u4fdd\u7559\u5468\u671f\u65f6\uff0c\u5efa\u8bae\u7ed3\u5408\u4e1a\u52a1\u9700\u6c42\u52a8\u6001\u8c03\u6574\u3002\u5bf9\u6838\u5fc3\u4e1a\u52a1\u6307\u6807\u4fdd\u755930\u5929\uff0c\u975e\u5173\u952e\u65e5\u5fd7\u7c7b\u6307\u6807\u4fdd\u75597\u5929\uff0c\u53ef\u51cf\u5c1140%\u4ee5\u4e0a\u7684\u5b58\u50a8\u7a7a\u95f4\u5360\u7528\u3002\n<\/li>\n
  3. \u5f15\u5165\u8fdc\u7a0b\u5b58\u50a8<\/strong>
    \n\u5f53\u5355\u673a\u5b58\u50a8\u8fbe\u5230\u74f6\u9888\u65f6\uff0cThanos<\/em>\u3001*Cortex*\u7b49\u89e3\u51b3\u65b9\u6848\u80fd\u5b9e\u73b0\u6570\u636e\u5206\u7247\u4e0e\u957f\u671f\u5b58\u50a8\u3002\u67d0\u7535\u5546\u5e73\u53f0\u901a\u8fc7Thanos\u5c06\u5386\u53f2\u6570\u636e\u8fc1\u79fb\u81f3\u5bf9\u8c61\u5b58\u50a8\u540e\uff0cPrometheus\u5b9e\u4f8b\u7684\u672c\u5730\u5b58\u50a8\u538b\u529b\u964d\u4f4e\u4e8675%\u3002\n<\/li>\n<\/ol>\n
    \n

    \u4e8c\u3001\u63d0\u5347\u67e5\u8be2\u6548\u7387\uff1a\u89e3\u51b3\u9ad8\u5ef6\u8fdf\u7684\u75c7\u7ed3<\/strong><\/h3>\n

    \u6162\u67e5\u8be2\u662f\u5bfc\u81f4Prometheus\u6027\u80fd\u95ee\u9898\u7684\u5e38\u89c1\u539f\u56e0\u3002\u4f18\u5316PromQL\u7f16\u5199\u4e0e\u67e5\u8be2\u7b56\u7565\uff0c\u53ef\u907f\u514d\u8d44\u6e90\u8fc7\u8f7d<\/strong>\uff1a<\/p>\n

      \n
    1. \u9650\u5236\u65f6\u95f4\u8303\u56f4\u4e0e\u5206\u8fa8\u7387<\/strong>
      \n\u907f\u514d\u5168\u91cf\u65f6\u95f4\u8303\u56f4\u67e5\u8be2\uff08\u5982metric{}<\/code>\uff09\uff0c\u6539\u7528metric[1h]<\/code>\u9650\u5b9a\u65f6\u95f4\u7a97\u53e3\u3002\u5bf9\u4e8e\u4eea\u8868\u76d8\u7c7b\u9700\u6c42\uff0c\u901a\u8fc7--query.max-samples<\/code>\u9650\u5236\u5355\u6b21\u67e5\u8be2\u7684\u6837\u672c\u6570\u91cf\u3002\n<\/li>\n
    2. \u9884\u8ba1\u7b97\u4e0e\u89c4\u5219\u4f18\u5316<\/strong>
      \n\u5c06\u9ad8\u9891\u67e5\u8be2\u7684\u8868\u8fbe\u5f0f\uff08\u5982rate(http_requests_total[5m])<\/code>\uff09\u9884\u5148\u5b9a\u4e49\u4e3aRecording Rules\uff0c\u53ef\u51cf\u5c11\u5b9e\u65f6\u8ba1\u7b97\u5f00\u9500\u3002\u67d0\u91d1\u878d\u7cfb\u7edf\u901a\u8fc7\u9884\u8ba1\u7b97\u6838\u5fc3\u6307\u6807\uff0c\u5c06P99\u67e5\u8be2\u5ef6\u8fdf\u4ece8\u79d2\u964d\u81f30.5\u79d2\u3002\n<\/li>\n
    3. \u89c4\u907f\u6b63\u5219\u8868\u8fbe\u5f0f\u9677\u9631<\/strong>
      \n\u5728\u6807\u7b7e\u5339\u914d\u65f6\uff0c\u4f18\u5148\u4f7f\u7528=<\/code>\u7cbe\u786e\u5339\u914d\u800c\u975e=~<\/code>\u6b63\u5219\u5339\u914d\u3002\u4f8b\u5982\uff0c{job="api-server"}<\/code>\u6bd4{job=~"api.*"}<\/code>\u6548\u7387\u63d0\u53473\u500d\u4ee5\u4e0a\u3002\n<\/li>\n<\/ol>\n
      \n

      \u4e09\u3001\u914d\u7f6e\u8c03\u4f18\uff1a\u7cbe\u7ec6\u5316\u63a7\u5236\u8d44\u6e90\u5206\u914d<\/strong><\/h3>\n

      \u4e0d\u5f53\u7684\u91c7\u96c6\u914d\u7f6e\u4f1a\u76f4\u63a5\u5f71\u54cdPrometheus\u6027\u80fd\u3002\u901a\u8fc7\u52a8\u6001\u8c03\u6574\u91c7\u96c6\u884c\u4e3a\uff0c\u53ef\u5b9e\u73b0\u8d44\u6e90\u5229\u7528\u7387\u6700\u5927\u5316<\/strong>\uff1a<\/p>\n

        \n
      1. \u5408\u7406\u8bbe\u7f6e\u91c7\u96c6\u9891\u7387<\/strong>
        \n\u6839\u636e\u6307\u6807\u91cd\u8981\u6027\u5206\u7ea7\u914d\u7f6escrape_interval<\/code>\u3002\u5173\u952e\u4e1a\u52a1\u6307\u6807\u91c7\u752815\u79d2\u95f4\u9694\uff0c\u8fd0\u7ef4\u7c7b\u6307\u6807\u53ef\u653e\u5bbd\u81f31\u5206\u949f\u3002\u67d0\u7269\u8054\u7f51\u5e73\u53f0\u901a\u8fc7\u5206\u7ea7\u91c7\u96c6\u7b56\u7565\uff0cCPU\u4f7f\u7528\u7387\u964d\u4f4e22%\u3002\n<\/li>\n
      2. \u4f18\u5316\u670d\u52a1\u53d1\u73b0\u673a\u5236<\/strong>
        \n\u5728Kubernetes\u73af\u5883\u4e2d\uff0c\u542f\u7528role: service<\/code>\u8fc7\u6ee4\u975e\u5fc5\u8981Endpoint\u3002\u4f7f\u7528relabel_configs<\/code>\u5220\u9664\u65e0\u5173\u6807\u7b7e\uff08\u5982__meta_kubernetes_pod_node_name<\/code>\uff09\uff0c\u53ef\u4f7f\u5185\u5b58\u5360\u7528\u51cf\u5c1110%-15%\u3002\n<\/li>\n
      3. \u6807\u7b7e\u7ba1\u7406\u9ec4\u91d1\u6cd5\u5219<\/strong>
        \n\u9075\u5faa\u201c\u6700\u5c0f\u5316\u6807\u7b7e\u57fa\u6570\u201d\u539f\u5219\uff0c\u907f\u514d\u9ad8\u57fa\u6570\u6807\u7b7e\uff08\u5982\u7528\u6237ID\uff09\u3002\u5f53\u67d0\u4e2a\u6307\u6807\u7684\u6807\u7b7e\u7ec4\u5408\u8d85\u8fc71\u4e07\u79cd\u65f6\uff0c\u5efa\u8bae\u62c6\u5206\u6216\u805a\u5408\u6307\u6807\u3002\n<\/li>\n<\/ol>\n
        \n

        \u56db\u3001\u786c\u4ef6\u4e0e\u67b6\u6784\u8bbe\u8ba1\uff1a\u7a81\u7834\u5355\u673a\u6027\u80fd\u74f6\u9888<\/strong><\/h3>\n

        \u5f53\u5355\u5b9e\u4f8b\u6027\u80fd\u8fbe\u5230\u6781\u9650\u65f6\uff0c\u6a2a\u5411\u6269\u5c55\u4e0e\u67b6\u6784\u4f18\u5316\u6210\u4e3a\u5173\u952e<\/strong>\uff1a<\/p>\n

          \n
        1. \u6c34\u5e73\u5206\u7247\u7b56\u7565<\/strong>
          \n\u6839\u636e\u4e1a\u52a1\u903b\u8f91\u5bf9Target\u8fdb\u884c\u5206\u7247\uff08Sharding\uff09\u3002\u4f8b\u5982\uff0c\u6309\u5730\u57df\u5c06\u670d\u52a1\u5668\u5212\u5206\u4e3ashard=us-east<\/code>\u548cshard=eu-west<\/code>\uff0c\u914d\u5408\u8d1f\u8f7d\u5747\u8861\u5b9e\u73b0\u5e76\u884c\u91c7\u96c6\u3002\n<\/li>\n
        2. \u8bfb\u5199\u5206\u79bb\u67b6\u6784<\/strong>
          \n\u4f7f\u7528Prometheus\u7684\u8fdc\u7a0b\u8bfb\u5199\u63a5\u53e3\uff0c\u5c06\u6570\u636e\u5199\u5165\u4e0e\u67e5\u8be2\u5206\u79bb\u3002\u67d0\u6e38\u620f\u516c\u53f8\u901a\u8fc7\u90e8\u7f72\u4e13\u7528\u67e5\u8be2\u8282\u70b9\uff0c\u4f7f\u524d\u7aef\u67e5\u8be2\u6027\u80fd\u63d0\u534760%\u3002\n<\/li>\n
        3. \u7f13\u5b58\u673a\u5236\u5e94\u7528<\/strong>
          \n\u5728\u67e5\u8be2\u5c42\u5f15\u5165Redis\u7f13\u5b58\u9ad8\u9891\u67e5\u8be2\u7ed3\u679c\u3002\u5bf9\u4e8esum by (instance) (cpu_usage)<\/code>\u7b49\u56fa\u5b9a\u62a5\u8868\u7c7b\u67e5\u8be2\uff0c\u8bbe\u7f6e5\u5206\u949f\u7f13\u5b58\u5468\u671f\u53ef\u964d\u4f4e50%\u7684\u91cd\u590d\u8ba1\u7b97\u3002\n<\/li>\n<\/ol>\n
          \n

          \u4e94\u3001\u5176\u4ed6\u5b9e\u6218\u6280\u5de7<\/strong><\/h3>\n
            \n
          • \u7248\u672c\u5347\u7ea7\u7b56\u7565<\/strong>\uff1a\u5b9a\u671f\u66f4\u65b0\u81f3\u7a33\u5b9a\u7248\u672c\uff08\u5982v2.40+\uff09\uff0c\u65b0\u7248TSDB\u7684\u5199\u5165\u541e\u5410\u91cf\u8f83\u65e9\u671f\u7248\u672c\u63d0\u53472\u500d\n<\/li>\n
          • \u5185\u5b58\u4f18\u5316\u6280\u5de7<\/strong>\uff1a\u901a\u8fc7--storage.tsdb.memory-chunks<\/code>\u9650\u5236\u5185\u5b58\u4e2d\u4fdd\u7559\u7684Chunk\u6570\u91cf\uff0c\u907f\u514dOOM\n<\/li>\n
          • \u76d1\u63a7\u81ea\u8eab\u72b6\u6001<\/strong>\uff1a\u6301\u7eed\u5173\u6ce8prometheus_tsdb_head_series<\/code>\u7b49\u5185\u7f6e\u6307\u6807\uff0c\u5f53Series\u6570\u91cf\u8d85\u8fc7200\u4e07\u65f6\u9700\u7acb\u5373\u6269\u5bb9\n<\/li>\n
          • \u65e5\u5fd7\u7ea7\u522b\u8c03\u6574<\/strong>\uff1a\u751f\u4ea7\u73af\u5883\u8bbe\u7f6e--log.level=error<\/code>\uff0c\u907f\u514d\u8c03\u8bd5\u65e5\u5fd7\u62d6\u6162\u6027\u80fd\n<\/li>\n<\/ul>\n

            \u901a\u8fc7\u4ee5\u4e0a\u7b56\u7565\u7684\u7ec4\u5408\u5e94\u7528\uff0c\u67d0\u5934\u90e8\u4e91\u670d\u52a1\u5546\u6210\u529f\u5c06Prometheus\u96c6\u7fa4\u7684\u65e5\u5747\u6570\u636e\u5904\u7406\u91cf\u4ece2TB\u63d0\u5347\u81f38TB\uff0c\u540c\u65f6\u4fdd\u6301P99\u67e5\u8be2\u5ef6\u8fdf\u4f4e\u4e8e500\u6beb\u79d2\u3002\u8fd9\u4e9b\u5b9e\u8df5\u8868\u660e\uff0c\u7cfb\u7edf\u5316\u7684\u6027\u80fd\u4f18\u5316\u4e0d\u4ec5\u80fd\u63d0\u5347\u76d1\u63a7\u6548\u7387\uff0c\u66f4\u80fd\u4e3a\u4e1a\u52a1\u51b3\u7b56\u63d0\u4f9b\u5b9e\u65f6\u3001\u53ef\u9760\u7684\u6570\u636e\u652f\u6491<\/strong>\u3002<\/p>\n","protected":false},"excerpt":{"rendered":"

            \u968f\u7740\u4e91\u539f\u751f\u6280\u672f\u7684\u5feb\u901f\u53d1\u5c55\uff0cPrometheus\u5df2\u6210\u4e3a\u76d1\u63a7\u9886\u57df\u7684\u6838\u5fc3\u5de5\u5177\u4e4b\u4e00\u3002\u7136\u800c\uff0c\u968f\u7740\u6570\u636e\u91cf\u7684\u6307\u6570\u7ea7\u589e\u957f\u548c\u5206\u5e03\u5f0f\u7cfb\u7edf\u7684\u590d\u6742\u6027\u63d0\u5347\uff0c\u8bb8\u591a\u7528\u6237\u53d1\u73b0\u9ed8\u8ba4\u914d\u7f6e\u4e0b\u7684Prometheus\u53ef\u80fd\u9762\u4e34\u67e5\u8be2\u5ef6\u8fdf\u9ad8\u3001\u5b58\u50a8\u538b\u529b\u5927\u7b49\u95ee\u9898\u3002\u5982\u4f55\u901a\u8fc7\u79d1\u5b66\u4f18\u5316\u7b56\u7565\u91ca\u653ePrometheus\u7684\u6f5c\u5728\u6027\u80fd\uff1f \u672c\u6587\u5c06\u4ece\u5b58\u50a8\u673a\u5236\u3001\u67e5\u8be2\u6548\u7387\u3001\u914d\u7f6e\u8c03\u4f18\u7b49\u591a\u4e2a\u89d2\u5ea6\uff0c\u6df1\u5165\u63a2\u8ba8\u63d0\u5347Prometheus\u6027\u80fd\u7684\u5b9e\u8df5\u65b9\u6cd5\u3002 \u4e00\u3001\u4f18\u5316\u5b58\u50a8\u673a\u5236\uff1a\u4ece\u6839\u6e90\u964d\u4f4e\u8d44\u6e90\u6d88\u8017 Prometheus\u7684TSDB\uff08\u65f6\u95f4\u5e8f\u5217\u6570\u636e\u5e93\uff09\u662f\u5176\u6027\u80fd\u8868\u73b0\u7684\u6838\u5fc3\u3002\u901a\u8fc7\u8c03\u6574\u5b58\u50a8\u7b56\u7565\uff0c\u53ef\u663e\u8457\u51cf\u5c11\u78c1\u76d8\u5360\u7528\u548c\u5185\u5b58\u538b\u529b\uff1a \u542f\u7528\u5757\u538b\u7f29\u4e0e\u5408\u5e76 TSDB\u9ed8\u8ba4\u6bcf2\u5c0f\u65f6\u5c06\u5185\u5b58\u4e2d\u7684\u6570\u636e\u5199\u5165\u78c1\u76d8\u5757\uff08block\uff09\u3002\u901a\u8fc7\u8c03\u6574–storage.tsdb.min-block-duration\u548c–storage.tsdb.max-block-duration\u53c2\u6570\uff0c\u53ef\u4ee5\u5e73\u8861\u6570\u636e\u5199\u5165\u9891\u7387\u4e0e\u67e5\u8be2\u6548\u7387\u3002\u4f8b\u5982\uff0c\u5728\u6570\u636e\u91cf\u6fc0\u589e\u7684\u73af\u5883\u4e2d\u5c06\u5757\u5468\u671f\u7f29\u77ed\u81f31\u5c0f\u65f6\uff0c\u53ef\u907f\u514d\u56e0\u5355\u5757\u8fc7\u5927\u5bfc\u81f4\u67e5\u8be2\u5ef6\u8fdf\u3002 \u7cbe\u7ec6\u5316\u4fdd\u7559\u7b56\u7565 \u4f7f\u7528–storage.tsdb.retention.time\u63a7\u5236\u6570\u636e\u4fdd\u7559\u5468\u671f\u65f6\uff0c\u5efa\u8bae\u7ed3\u5408\u4e1a\u52a1\u9700\u6c42\u52a8\u6001\u8c03\u6574\u3002\u5bf9\u6838\u5fc3\u4e1a\u52a1\u6307\u6807\u4fdd\u755930\u5929\uff0c\u975e\u5173\u952e\u65e5\u5fd7\u7c7b\u6307\u6807\u4fdd\u75597\u5929\uff0c\u53ef\u51cf\u5c1140%\u4ee5\u4e0a\u7684\u5b58\u50a8\u7a7a\u95f4\u5360\u7528\u3002 \u5f15\u5165\u8fdc\u7a0b\u5b58\u50a8 \u5f53\u5355\u673a\u5b58\u50a8\u8fbe\u5230\u74f6\u9888\u65f6\uff0cThanos\u3001*Cortex*\u7b49\u89e3\u51b3\u65b9\u6848\u80fd\u5b9e\u73b0\u6570\u636e\u5206\u7247\u4e0e\u957f\u671f\u5b58\u50a8\u3002\u67d0\u7535\u5546\u5e73\u53f0\u901a\u8fc7Thanos\u5c06\u5386\u53f2\u6570\u636e\u8fc1\u79fb\u81f3\u5bf9\u8c61\u5b58\u50a8\u540e\uff0cPrometheus\u5b9e\u4f8b\u7684\u672c\u5730\u5b58\u50a8\u538b\u529b\u964d\u4f4e\u4e8675%\u3002 \u4e8c\u3001\u63d0\u5347\u67e5\u8be2\u6548\u7387\uff1a\u89e3\u51b3\u9ad8\u5ef6\u8fdf\u7684\u75c7\u7ed3 \u6162\u67e5\u8be2\u662f\u5bfc\u81f4Prometheus\u6027\u80fd\u95ee\u9898\u7684\u5e38\u89c1\u539f\u56e0\u3002\u4f18\u5316PromQL\u7f16\u5199\u4e0e\u67e5\u8be2\u7b56\u7565\uff0c\u53ef\u907f\u514d\u8d44\u6e90\u8fc7\u8f7d\uff1a \u9650\u5236\u65f6\u95f4\u8303\u56f4\u4e0e\u5206\u8fa8\u7387 \u907f\u514d\u5168\u91cf\u65f6\u95f4\u8303\u56f4\u67e5\u8be2\uff08\u5982metric{}\uff09\uff0c\u6539\u7528metric[1h]\u9650\u5b9a\u65f6\u95f4\u7a97\u53e3\u3002\u5bf9\u4e8e\u4eea\u8868\u76d8\u7c7b\u9700\u6c42\uff0c\u901a\u8fc7–query.max-samples\u9650\u5236\u5355\u6b21\u67e5\u8be2\u7684\u6837\u672c\u6570\u91cf\u3002 \u9884\u8ba1\u7b97\u4e0e\u89c4\u5219\u4f18\u5316 \u5c06\u9ad8\u9891\u67e5\u8be2\u7684\u8868\u8fbe\u5f0f\uff08\u5982rate(http_requests_total[5m])\uff09\u9884\u5148\u5b9a\u4e49\u4e3aRecording Rules\uff0c\u53ef\u51cf\u5c11\u5b9e\u65f6\u8ba1\u7b97\u5f00\u9500\u3002\u67d0\u91d1\u878d\u7cfb\u7edf\u901a\u8fc7\u9884\u8ba1\u7b97\u6838\u5fc3\u6307\u6807\uff0c\u5c06P99\u67e5\u8be2\u5ef6\u8fdf\u4ece8\u79d2\u964d\u81f30.5\u79d2\u3002 \u89c4\u907f\u6b63\u5219\u8868\u8fbe\u5f0f\u9677\u9631 \u5728\u6807\u7b7e\u5339\u914d\u65f6\uff0c\u4f18\u5148\u4f7f\u7528=\u7cbe\u786e\u5339\u914d\u800c\u975e=~\u6b63\u5219\u5339\u914d\u3002\u4f8b\u5982\uff0c{job="api-server"}\u6bd4{job=~"api.*"}\u6548\u7387\u63d0\u53473\u500d\u4ee5\u4e0a\u3002 \u4e09\u3001\u914d\u7f6e\u8c03\u4f18\uff1a\u7cbe\u7ec6\u5316\u63a7\u5236\u8d44\u6e90\u5206\u914d \u4e0d\u5f53\u7684\u91c7\u96c6\u914d\u7f6e\u4f1a\u76f4\u63a5\u5f71\u54cdPrometheus\u6027\u80fd\u3002\u901a\u8fc7\u52a8\u6001\u8c03\u6574\u91c7\u96c6\u884c\u4e3a\uff0c\u53ef\u5b9e\u73b0\u8d44\u6e90\u5229\u7528\u7387\u6700\u5927\u5316\uff1a \u5408\u7406\u8bbe\u7f6e\u91c7\u96c6\u9891\u7387 \u6839\u636e\u6307\u6807\u91cd\u8981\u6027\u5206\u7ea7\u914d\u7f6escrape_interval\u3002\u5173\u952e\u4e1a\u52a1\u6307\u6807\u91c7\u752815\u79d2\u95f4\u9694\uff0c\u8fd0\u7ef4\u7c7b\u6307\u6807\u53ef\u653e\u5bbd\u81f31\u5206\u949f\u3002\u67d0\u7269\u8054\u7f51\u5e73\u53f0\u901a\u8fc7\u5206\u7ea7\u91c7\u96c6\u7b56\u7565\uff0cCPU\u4f7f\u7528\u7387\u964d\u4f4e22%\u3002 \u4f18\u5316\u670d\u52a1\u53d1\u73b0\u673a\u5236 \u5728Kubernetes\u73af\u5883\u4e2d\uff0c\u542f\u7528role: service\u8fc7\u6ee4\u975e\u5fc5\u8981Endpoint\u3002\u4f7f\u7528relabel_configs\u5220\u9664\u65e0\u5173\u6807\u7b7e\uff08\u5982__meta_kubernetes_pod_node_name\uff09\uff0c\u53ef\u4f7f\u5185\u5b58\u5360\u7528\u51cf\u5c1110%-15%\u3002 \u6807\u7b7e\u7ba1\u7406\u9ec4\u91d1\u6cd5\u5219 \u9075\u5faa\u201c\u6700\u5c0f\u5316\u6807\u7b7e\u57fa\u6570\u201d\u539f\u5219\uff0c\u907f\u514d\u9ad8\u57fa\u6570\u6807\u7b7e\uff08\u5982\u7528\u6237ID\uff09\u3002\u5f53\u67d0\u4e2a\u6307\u6807\u7684\u6807\u7b7e\u7ec4\u5408\u8d85\u8fc71\u4e07\u79cd\u65f6\uff0c\u5efa\u8bae\u62c6\u5206\u6216\u805a\u5408\u6307\u6807\u3002 \u56db\u3001\u786c\u4ef6\u4e0e\u67b6\u6784\u8bbe\u8ba1\uff1a\u7a81\u7834\u5355\u673a\u6027\u80fd\u74f6\u9888 \u5f53\u5355\u5b9e\u4f8b\u6027\u80fd\u8fbe\u5230\u6781\u9650\u65f6\uff0c\u6a2a\u5411\u6269\u5c55\u4e0e\u67b6\u6784\u4f18\u5316\u6210\u4e3a\u5173\u952e\uff1a \u6c34\u5e73\u5206\u7247\u7b56\u7565 \u6839\u636e\u4e1a\u52a1\u903b\u8f91\u5bf9Target\u8fdb\u884c\u5206\u7247\uff08Sharding\uff09\u3002\u4f8b\u5982\uff0c\u6309\u5730\u57df\u5c06\u670d\u52a1\u5668\u5212\u5206\u4e3ashard=us-east\u548cshard=eu-west\uff0c\u914d\u5408\u8d1f\u8f7d\u5747\u8861\u5b9e\u73b0\u5e76\u884c\u91c7\u96c6\u3002 \u8bfb\u5199\u5206\u79bb\u67b6\u6784 \u4f7f\u7528Prometheus\u7684\u8fdc\u7a0b\u8bfb\u5199\u63a5\u53e3\uff0c\u5c06\u6570\u636e\u5199\u5165\u4e0e\u67e5\u8be2\u5206\u79bb\u3002\u67d0\u6e38\u620f\u516c\u53f8\u901a\u8fc7\u90e8\u7f72\u4e13\u7528\u67e5\u8be2\u8282\u70b9\uff0c\u4f7f\u524d\u7aef\u67e5\u8be2\u6027\u80fd\u63d0\u534760%\u3002 \u7f13\u5b58\u673a\u5236\u5e94\u7528 \u5728\u67e5\u8be2\u5c42\u5f15\u5165Redis\u7f13\u5b58\u9ad8\u9891\u67e5\u8be2\u7ed3\u679c\u3002\u5bf9\u4e8esum by (instance) (cpu_usage)\u7b49\u56fa\u5b9a\u62a5\u8868\u7c7b\u67e5\u8be2\uff0c\u8bbe\u7f6e5\u5206\u949f\u7f13\u5b58\u5468\u671f\u53ef\u964d\u4f4e50%\u7684\u91cd\u590d\u8ba1\u7b97\u3002 \u4e94\u3001\u5176\u4ed6\u5b9e\u6218\u6280\u5de7 \u7248\u672c\u5347\u7ea7\u7b56\u7565\uff1a\u5b9a\u671f\u66f4\u65b0\u81f3\u7a33\u5b9a\u7248\u672c\uff08\u5982v2.40+\uff09\uff0c\u65b0\u7248TSDB\u7684\u5199\u5165\u541e\u5410\u91cf\u8f83\u65e9\u671f\u7248\u672c\u63d0\u53472\u500d \u5185\u5b58\u4f18\u5316\u6280\u5de7\uff1a\u901a\u8fc7–storage.tsdb.memory-chunks\u9650\u5236\u5185\u5b58\u4e2d\u4fdd\u7559\u7684Chunk\u6570\u91cf\uff0c\u907f\u514dOOM \u76d1\u63a7\u81ea\u8eab\u72b6\u6001\uff1a\u6301\u7eed\u5173\u6ce8prometheus_tsdb_head_series\u7b49\u5185\u7f6e\u6307\u6807\uff0c\u5f53Series\u6570\u91cf\u8d85\u8fc7200\u4e07\u65f6\u9700\u7acb\u5373\u6269\u5bb9 \u65e5\u5fd7\u7ea7\u522b\u8c03\u6574\uff1a\u751f\u4ea7\u73af\u5883\u8bbe\u7f6e–log.level=error\uff0c\u907f\u514d\u8c03\u8bd5\u65e5\u5fd7\u62d6\u6162\u6027\u80fd \u901a\u8fc7\u4ee5\u4e0a\u7b56\u7565\u7684\u7ec4\u5408\u5e94\u7528\uff0c\u67d0\u5934\u90e8\u4e91\u670d\u52a1\u5546\u6210\u529f\u5c06Prometheus\u96c6\u7fa4\u7684\u65e5\u5747\u6570\u636e\u5904\u7406\u91cf\u4ece2TB\u63d0\u5347\u81f38TB\uff0c\u540c\u65f6\u4fdd\u6301P99\u67e5\u8be2\u5ef6\u8fdf\u4f4e\u4e8e500\u6beb\u79d2\u3002\u8fd9\u4e9b\u5b9e\u8df5\u8868\u660e\uff0c\u7cfb\u7edf\u5316\u7684\u6027\u80fd\u4f18\u5316\u4e0d\u4ec5\u80fd\u63d0\u5347\u76d1\u63a7\u6548\u7387\uff0c\u66f4\u80fd\u4e3a\u4e1a\u52a1\u51b3\u7b56\u63d0\u4f9b\u5b9e\u65f6\u3001\u53ef\u9760\u7684\u6570\u636e\u652f\u6491\u3002<\/p>\n","protected":false},"author":4,"featured_media":35432,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[14],"tags":[],"class_list":["post-35431","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-product-news"],"_links":{"self":[{"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/posts\/35431","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/users\/4"}],"replies":[{"embeddable":true,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/comments?post=35431"}],"version-history":[{"count":1,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/posts\/35431\/revisions"}],"predecessor-version":[{"id":35433,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/posts\/35431\/revisions\/35433"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/media\/35432"}],"wp:attachment":[{"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/media?parent=35431"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/categories?post=35431"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.yunshan.net\/blog\/wp-json\/wp\/v2\/tags?post=35431"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}