{"id":1026,"date":"2022-01-20T11:37:59","date_gmt":"2022-01-20T03:37:59","guid":{"rendered":"https:\/\/aichh.com\/?p=1026"},"modified":"2022-01-20T11:41:06","modified_gmt":"2022-01-20T03:41:06","slug":"centos-bug%ef%bc%9asoft-lockup-cpux","status":"publish","type":"post","link":"https:\/\/aichh.com\/1026.html","title":{"rendered":"[CentOS] BUG\uff1asoft lockup &#8211;CPU#x"},"content":{"rendered":"<!--wp-compress-html--><!--wp-compress-html no compression--><p>\u7531\u4e8eZabbix\u4e2d\u7ecf\u5e38\u51fa\u73b0IO\u7cfb\u7edf\u9519\u8bef\uff0c\u6240\u4ee5\u5728\u67e5\u770b\u65f6\u53d1\u73b0\u9519\u8bef<strong>\u5185\u6838\uff1aBUG: soft lockup --CPU#x<\/strong>\uff0c\u6240\u4ee5\u5904\u7406\u7684\u65f6\u5019\u505a\u4e2a\u5907\u5fd8\u3002<\/p>\n<h2>\u73af\u5883<\/h2>\n<ul>\n<li>ESXi 6.0<\/li>\n<li>CentOS 7.2<\/li>\n<\/ul>\n<h2>\u9519\u8bef\u7b49<\/h2>\n<p>\u5f53\u6211\u68c0\u67e5\/var\/log\/messages\u65f6\uff0c\u8f93\u51fa\u4e86\u4ee5\u4e0b\u9519\u8bef\u3002<\/p>\n<pre>Jun 29 03:30:38 HOSTNAME kernel: BUG: soft lockup - CPU#1 stuck for 23s! [xfsaild\/dm-0:401]\r\nJun 29 03:30:38 HOSTNAME kernel: Modules linked in: ip6t_rpfilter ipt_REJECT ip6t_REJECT xt_conntrack ip_set nfnetlink ebtable_nat ebtable_broute bridge stp llc ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6table_mangle ip6table_security ip6table_raw iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack iptable_mangle iptable_security iptable_raw ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter vmw_vsock_vmci_transport vsock coretemp crc32_pclmul ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd pcspkr ppdev sg vmw_balloon parport_pc parport shpchp i2c_piix4 vmw_vmci ip_tables xfs libcrc32c sr_mod cdrom sd_mod crc_t10dif crct10dif_generic ata_generic pata_acpi crct10dif_pclmul crct10dif_common crc32c_intel serio_raw vmwgfx e1000 drm_kms_helper\r\nJun 29 03:30:38 HOSTNAME kernel: ttm mptspi scsi_transport_spi mptscsih drm mptbase ata_piix i2c_core libata floppy dm_mirror dm_region_hash dm_log dm_mod\r\nJun 29 03:30:38 HOSTNAME kernel: CPU: 1 PID: 401 Comm: xfsaild\/dm-0 Tainted: G             L ------------   3.10.0-327.el7.x86_64 #1\r\nJun 29 03:30:38 HOSTNAME kernel: Hardware name: VMware, Inc. VMware Virtual Platform\/440BX Desktop Reference Platform, BIOS 6.00 09\/30\/2014\r\nJun 29 03:30:38 HOSTNAME kernel: task: ffff880135d96780 ti: ffff880036380000 task.ti: ffff880036380000\r\nJun 29 03:30:38 HOSTNAME kernel: RIP: 0010:[]  [] mpt_put_msg_frame+0x5e\/0x80 [mptbase]\r\nJun 29 03:30:38 HOSTNAME kernel: RSP: 0018:ffff880036383b00  EFLAGS: 00000246\r\nJun 29 03:30:38 HOSTNAME kernel: RAX: ffffc90008780000 RBX: ffff880036383ad0 RCX: 0000000000000018\r\nJun 29 03:30:38 HOSTNAME kernel: RDX: ffff880135c4f600 RSI: ffff880139157000 RDI: 000000000000000e\r\nJun 29 03:30:38 HOSTNAME kernel: RBP: ffff880036383b10 R08: 0000000000000002 R09: ffff88003640b0d8\r\nJun 29 03:30:38 HOSTNAME kernel: R10: ffff8800ba2c16c0 R11: ffff8800ba2c16c0 R12: 0000000100000001\r\nJun 29 03:30:38 HOSTNAME kernel: R13: ffff880097c20300 R14: ffff880036383a74 R15: ffffffff812d06bb\r\nJun 29 03:30:38 HOSTNAME kernel: FS:  0000000000000000(0000) GS:ffff88013fd00000(0000) knlGS:0000000000000000\r\nJun 29 03:30:38 HOSTNAME kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b\r\nJun 29 03:30:38 HOSTNAME kernel: CR2: 00007fc2c650f000 CR3: 00000000ba1e0000 CR4: 00000000000407e0\r\nJun 29 03:30:38 HOSTNAME kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000\r\nJun 29 03:30:38 HOSTNAME kernel: DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400\r\nJun 29 03:30:38 HOSTNAME kernel: Stack:\r\nJun 29 03:30:38 HOSTNAME kernel: ffff880139157d30 000000000000003c ffff880036383bb8 ffffffffa008b729\r\nJun 29 03:30:38 HOSTNAME kernel: ffff880139157008 0400000036383b78 0000000000000018 ffff8800ba2c16c0\r\nJun 29 03:30:38 HOSTNAME kernel: 0000000000000060 ffff880139157188 0000006000000018 ffff880036603130\r\nJun 29 03:30:38 HOSTNAME kernel: Call Trace:\r\nJun 29 03:30:38 HOSTNAME kernel: [] mptscsih_qcmd+0x249\/0x820 [mptscsih]\r\nJun 29 03:30:38 HOSTNAME kernel: [] mptspi_qcmd+0x50\/0xe0 [mptspi]\r\nJun 29 03:30:38 HOSTNAME kernel: [] scsi_dispatch_cmd+0xaa\/0x230\r\nJun 29 03:30:38 HOSTNAME kernel: [] scsi_request_fn+0x501\/0x770\r\nJun 29 03:30:38 HOSTNAME kernel: [] __blk_run_queue+0x33\/0x40\r\nJun 29 03:30:38 HOSTNAME kernel: [] queue_unplugged+0x2a\/0xa0\r\nJun 29 03:30:38 HOSTNAME kernel: [] blk_flush_plug_list+0x1d8\/0x230\r\nJun 29 03:30:38 HOSTNAME kernel: [] blk_finish_plug+0x14\/0x40\r\nJun 29 03:30:38 HOSTNAME kernel: [] __xfs_buf_delwri_submit+0x1e9\/0x250 [xfs]\r\nJun 29 03:30:38 HOSTNAME kernel: [] ? xfs_buf_delwri_submit_nowait+0x2f\/0x50 [xfs]\r\nJun 29 03:30:38 HOSTNAME kernel: [] ? xfs_trans_ail_cursor_first+0x90\/0x90 [xfs]\r\nJun 29 03:30:38 HOSTNAME kernel: [] xfs_buf_delwri_submit_nowait+0x2f\/0x50 [xfs]\r\nJun 29 03:30:38 HOSTNAME kernel: [] xfsaild+0x240\/0x5e0 [xfs]\r\nJun 29 03:30:38 HOSTNAME kernel: [] ? xfs_trans_ail_cursor_first+0x90\/0x90 [xfs]\r\nJun 29 03:30:38 HOSTNAME kernel: [] kthread+0xcf\/0xe0\r\nJun 29 03:30:38 HOSTNAME kernel: [] ? kthread_create_on_node+0x140\/0x140\r\nJun 29 03:30:38 HOSTNAME kernel: [] ret_from_fork+0x58\/0x90\r\nJun 29 03:30:38 HOSTNAME kernel: [] ? kthread_create_on_node+0x140\/0x140\r\nJun 29 03:30:38 HOSTNAME kernel: Code: 8b 96 68 01 00 00 0f b7 c8 44 03 a6 b0 01 00 00 44 8b 04 8a 45 09 c4 f686 e0 00 00 00 04 75 10 48 8b 83 e8 00 00 00 44 89 60 40 &amp;lt;5b&amp;gt; 41 5c 5d c3 48 8d 76 08 0f b7 c8 44 89 e2 48 c7 c7 78 c3 0c\r\n<\/pre>\n<p>&nbsp;<\/p>\n<p>\u7531\u4e8e\u662f\u534a\u591c\u53d1\u751f\u7684\uff0c\u6240\u4ee5\u6ca1\u6709\u4ec0\u4e48\u7279\u522b\u7684\u6548\u679c\uff0c\u4f46\u662f\u6211\u7528zabbix\u67e5\u770b\u76d1\u63a7\u72b6\u6001\u65f6\uff0c\u7ecf\u5e38\u51fa\u73b0IO\u76f8\u5173\u7684\u9519\u8bef\u3002\u7136\u800c\uff0c\u5b83\u5728\u51e0\u5206\u949f\u5185\u5b89\u5b9a\u4e0b\u6765\uff0c\u6062\u590d\u4e86\u539f\u6765\u7684\u72b6\u6001\u3002<\/p>\n<p>\u7ecf\u68c0\u67e5\uff0cVMware \u7684\u77e5\u8bc6\u5e93<a href=\"https:\/\/kb.vmware.com\/selfservice\/search.do?cmd=displayKC&amp;docType=kc&amp;docTypeID=DT_KB_1_1&amp;externalId=2094326\" target=\"_blank\" rel=\"noopener noreferrer\">\u6587\u7ae0\u201cA soft lockup message is output from a Linux kernel running on an SMP-enabled virtual machine (2094326)\u201d<\/a>\u6307\u51fa\uff1a<\/p>\n<div class=\"inyoumodoki\">\n<p>\u00a0Linux \u5185\u6838\u4e3a\u8f6f\u9501\u5b9a\u63d0\u4f9b\u4e86\u4e00\u4e2a\u770b\u95e8\u72d7\u7ebf\u7a0b\uff0c\u5e76\u4e14\u8fd9\u4e2a\u770b\u95e8\u72d7\u7ebf\u7a0b\u8c03\u5ea6\u8d85\u8fc7 10 \u79d2\uff0c\u5219\u4f1a\u6253\u5370\u4e00\u6761\u8f6f\u9501\u5b9a\u6d88\u606f\u3002<\/p>\n<\/div>\n<p>\u4f5c\u4e3a\u5bf9\u7b56\uff0c\u6709\u5982\u4e0b\u63cf\u8ff0\uff0c\u4f46\u6211\u770b\u5230\u8fd9\u6d88\u606f\u8fd8\u662f\u4e0d\u8212\u670d\uff0c\u6240\u4ee5\u6211\u60f3\u5904\u7406\u5b83\u3002..\u00a0..<\/p>\n<div class=\"inyoumodoki\">\n<p>\u8f6f\u9501\u5b9a\u6d88\u606f\u4e0d\u662f\u5185\u6838\u6545\u969c\uff0c\u53ef\u4ee5\u5b89\u5168\u5730\u5ffd\u7565\u3002<\/p>\n<\/div>\n<p>\u4e0a\u9762\u7684VM\u9875\u9762\u663e\u793a\u4e86\u5982\u4f55\u66f4\u6539\u8f6f\u9501\u5b9a\u9608\u503c\uff0c\u4f46\u8fd9\u4e0d\u662f\u6700\u7ec8\u7684\u89e3\u51b3\u65b9\u6848\uff1f\u5f53\u6211\u641c\u7d22\u5b83\u65f6\uff0c\u4e00\u4f4d\u4f3c\u4e4e\u662f\u4e13\u5bb6\u7684\u4eba\u89e3\u91ca\u8bf4<a href=\"https:\/\/groups.google.com\/forum\/#!topic\/jlug.ml.debian.users\/3Kus-uw43sA\" target=\"_blank\" rel=\"noopener noreferrer\">\u201cBUG\uff1asoft lockup --stops at CPU #\u201d \u3002<\/a><\/p>\n<p>\u5728VM\u73af\u5883overcommit\u7684\u60c5\u51b5\u4e0b\uff0cVM\u5e76\u4e0d\u603b\u662f\u5360\u7528\u771f\u6b63\u7684CPU\uff08\u5f53\u5176\u4ed6VM\u5fd9\u65f6\uff0c\u81ea\u7136\u6ca1\u6709\u8d44\u6e90\u5206\u914d\u7ed9\u7a7a\u95f2VM\uff09 \u8d44\u6e90\u6c60 \u9664\u975e\u5b83\u88ab\u63a7\u5236\u3002<\/p>\n<p>\u5f53\u7136\uff0c\u663e\u793a\u6d88\u606f\u7684\u73af\u5883\u662f\u4e00\u4e2a\u4e0d\u9519\u7684\u8fc7\u5ea6\u4f7f\u7528\u6761\u4ef6\u3002\u4ece\u7ecf\u9a8c\u6765\u770b\uff0c\u5185\u5b58\u5bf9\u8fc7\u5ea6\u4f7f\u7528\u6709\u70b9\u7d27\u5f20\uff0c\u56e0\u4e3a\u5f88\u5bb9\u6613\u901a\u8fc7\u4e00\u4e9b\u5f00\u9500\u770b\u5230\u6027\u80fd\u5dee\u5f02\uff0c\u4f46\u662f CPU \u8ba1\u7b97\u975e\u5e38\u56f0\u96be\uff0c\u56e0\u4e3a\u5e76\u975e\u6240\u6709 VM \u90fd\u5fd9\uff08\u7ecf\u5e38\uff09\u3002\u4e0d\u8981\uff08\u7ecf\u5e38\uff0c\u6211\uff09\u3002<\/p>\n<h2>\u5904\u7406<\/h2>\n<p>\u4f5c\u4e3a\u53c2\u8003\uff0c\u5982\u4e0b\u6240\u793a\u3002<\/p>\n<pre>\r\n# vi \/boot\/config-3.10.0-327.el7.x86_64\r\nCONFIG_LOCKUP_DETECTOR=y\r\n\u2193\r\nCONFIG_LOCKUP_DETECTOR=n\r\n<\/pre>\n<div class=\"crayon-toolbar\" data-settings=\" show\"><span class=\"crayon-title\">\u91cd\u542f<\/span><\/p>\n<pre>\r\n# shutdown -r now\r\n<\/pre>\n<p>\u5c31\u8fd9\u6837\u3002<\/p><\/div>\n<!--wp-compress-html no compression--><!--wp-compress-html-->","protected":false},"excerpt":{"rendered":"\u7531\u4e8eZabbix\u4e2d\u7ecf\u5e38\u51fa\u73b0IO\u7cfb\u7edf\u9519\u8bef\uff0c\u6240\u4ee5\u5728\u67e5\u770b\u65f6\u53d1\u73b0\u9519\u8bef\u5185\u6838\uff1aBUG: soft lockup &#8211;CPU#x\uff0c\u6240\u4ee5\u5904\u7406\u7684\u65f6\u5019\u505a\u4e2a\u5907\u5fd8\u3002 \u73af\u5883 ESXi 6.0 CentOS 7.2 \u9519\u8bef\u7b49 \u5f53\u6211 \u00b7\u00b7\u00b7","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3],"tags":[267,269,268],"class_list":["post-1026","post","type-post","status-publish","format-standard","hentry","category-jiaocheng","tag-bug","tag-lockup","tag-soft"],"views":405,"_links":{"self":[{"href":"https:\/\/aichh.com\/api\/wp\/v2\/posts\/1026","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/aichh.com\/api\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/aichh.com\/api\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/aichh.com\/api\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/aichh.com\/api\/wp\/v2\/comments?post=1026"}],"version-history":[{"count":5,"href":"https:\/\/aichh.com\/api\/wp\/v2\/posts\/1026\/revisions"}],"predecessor-version":[{"id":1031,"href":"https:\/\/aichh.com\/api\/wp\/v2\/posts\/1026\/revisions\/1031"}],"wp:attachment":[{"href":"https:\/\/aichh.com\/api\/wp\/v2\/media?parent=1026"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/aichh.com\/api\/wp\/v2\/categories?post=1026"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/aichh.com\/api\/wp\/v2\/tags?post=1026"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}