Skip to content

Instantly share code, notes, and snippets.

@mfenniak
Created May 23, 2017 14:00
Show Gist options
  • Save mfenniak/56f050b9a4877d1b07388e4d01de829c to your computer and use it in GitHub Desktop.
Save mfenniak/56f050b9a4877d1b07388e4d01de829c to your computer and use it in GitHub Desktop.
4.4.51-40.58.amzn1.x86_64 ena output
May 17 06:30:26 ip-172-16-15-126 consul[13099]: 2017/05/17 06:30:26 [INFO] serf: attempting reconnect to i-0ffe3ee8de2922539 172.16.14.98:8301
May 17 06:30:46 ip-172-16-15-126 consul[13099]: 2017/05/17 06:30:46 [INFO] agent: Synced node info
May 17 06:31:35 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 109170ms.
May 17 06:31:39 ip-172-16-15-126 consul[13099]: 2017/05/17 06:31:39 [WARN] memberlist: Refuting a suspect message (from: i-09ee29f77300a149d)
May 17 06:32:45 ip-172-16-15-126 consul[13099]: 2017/05/17 06:32:45 [INFO] serf: EventMemberJoin: i-03549fcaa8f4e1636 172.16.15.66
May 17 06:32:49 ip-172-16-15-126 consul[13099]: 2017/05/17 06:32:49 [INFO] memberlist: Marking i-057b08569e51a333d as failed, suspect timeout reached
May 17 06:32:49 ip-172-16-15-126 consul[13099]: 2017/05/17 06:32:49 [INFO] serf: EventMemberLeave: i-057b08569e51a333d 172.16.16.105
May 17 06:32:53 ip-172-16-15-126 consul[13099]: 2017/05/17 06:32:53 [WARN] memberlist: Refuting a suspect message (from: i-025dd16dcdd9a82c7)
May 17 06:33:15 ip-172-16-15-126 consul[13099]: 2017/05/17 06:33:15 [ERR] memberlist: failed to receive: EOF from=172.16.16.104:50079
May 17 06:33:15 ip-172-16-15-126 consul[13099]: 2017/05/17 06:33:15 [WARN] memberlist: Refuting a suspect message (from: i-097afca0ba61c8d02)
May 17 06:33:24 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 116940ms.
May 17 06:33:28 ip-172-16-15-126 consul[13099]: 2017/05/17 06:33:28 [ERR] memberlist: failed to receive: EOF from=172.16.17.196:49281
May 17 06:33:28 ip-172-16-15-126 consul[13099]: 2017/05/17 06:33:28 [WARN] memberlist: Refuting a suspect message (from: i-025dd16dcdd9a82c7)
May 17 06:34:22 ip-172-16-15-126 consul[13099]: 2017/05/17 06:34:22 [INFO] memberlist: Suspect i-0773de01abcf24ae9 has failed, no acks received
May 17 06:34:23 ip-172-16-15-126 consul[13099]: 2017/05/17 06:34:23 [INFO] memberlist: Suspect i-055f50877f84f0725 has failed, no acks received
May 17 06:34:24 ip-172-16-15-126 consul[13099]: 2017/05/17 06:34:24 [INFO] memberlist: Suspect i-0edd0cf9d6f372a8c has failed, no acks received
May 17 06:34:25 ip-172-16-15-126 consul[13099]: 2017/05/17 06:34:25 [INFO] memberlist: Suspect i-0a8cace27cc324e0f has failed, no acks received
May 17 06:34:26 ip-172-16-15-126 consul[13099]: 2017/05/17 06:34:26 [INFO] memberlist: Suspect i-0bd3ab67982f1fe55 has failed, no acks received
May 17 06:34:27 ip-172-16-15-126 consul[13099]: 2017/05/17 06:34:27 [INFO] memberlist: Suspect i-095ff5d9c84db7aed has failed, no acks received
May 17 06:35:21 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 128890ms.
May 17 06:37:30 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 110280ms.
May 17 06:39:21 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 123010ms.
May 17 06:41:24 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 114970ms.
May 17 06:43:19 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 124270ms.
May 17 06:45:23 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 117460ms.
May 17 06:47:21 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 124890ms.
May 17 06:49:25 ip-172-16-15-126 dhclient[3332]: XMT: Solicit on eth0, interval 112970ms.
May 17 06:50:19 ip-172-16-15-126 kernel: [365453.512122] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 613.
May 17 06:50:19 ip-172-16-15-126 kernel: [365453.516180] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 614.
May 17 06:50:19 ip-172-16-15-126 kernel: [365495.521705] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 818.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.492061] INFO: task jbd2/xvda1-8:2324 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.496712] Tainted: G E 4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.501576] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.507470] jbd2/xvda1-8 D ffff881e70553b08 0 2324 2 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.512832] ffff881e70553b08 ffff881e77ab0000 ffff881e70eb57c0 ffff881e70554000
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.519031] 0000000000000000 7fffffffffffffff ffff881efff9a0b8 ffffffff814ecd30
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.525187] ffff881e70553b20 ffffffff814ec5f5 ffff881e84695340 ffff881e70553bb8
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.531344] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.533441] [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.537239] [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.540853] [<ffffffff814eef01>] schedule_timeout+0x231/0x2b0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.544944] [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.549709] [<ffffffff810d2f7a>] ? ktime_get+0x3a/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.553585] [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.557396] [<ffffffff814ebcf6>] io_schedule_timeout+0xa6/0x110
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.561784] [<ffffffff814ecd4b>] bit_wait_io+0x1b/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.565713] [<ffffffff814ec990>] __wait_on_bit+0x60/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.569825] [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.573634] [<ffffffff814eca32>] out_of_line_wait_on_bit+0x72/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.578173] [<ffffffff810acf20>] ? autoremove_wake_function+0x40/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.582901] [<ffffffff812088c7>] __wait_on_buffer+0x27/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.587066] [<ffffffffa00cf2f6>] jbd2_journal_commit_transaction+0xf66/0x1bb0 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.592915] [<ffffffffa00d3c90>] kjournald2+0x120/0x340 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.597189] [<ffffffff810acee0>] ? prepare_to_wait_event+0xf0/0xf0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.601767] [<ffffffffa00d3b70>] ? commit_timeout+0x10/0x10 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.606347] [<ffffffff8108a7f9>] kthread+0xc9/0xe0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.610010] [<ffffffff8108a730>] ? kthread_park+0x60/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.614071] [<ffffffff814f02cf>] ret_from_fork+0x3f/0x70
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.618088] [<ffffffff8108a730>] ? kthread_park+0x60/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.622168] INFO: task dmeventd:3049 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.626981] Tainted: G E 4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.631856] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.637761] dmeventd D ffff881e728db8c8 0 3049 1 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.642896] ffff881e728db8c8 ffff881e77a99d40 ffff881e73598000 ffff881e728dc000
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.648990] 0000000000000000 7fffffffffffffff ffff881efff9a478 ffffffff814ecd30
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.655237] ffff881e728db8e0 ffffffff814ec5f5 ffff881e84635340 ffff881e728db978
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.661486] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.663586] [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.667459] [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.671223] [<ffffffff814eef01>] schedule_timeout+0x231/0x2b0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.675580] [<ffffffff8135f659>] ? notify_remote_via_irq+0x29/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.680217] [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.685124] [<ffffffff810d2f7a>] ? ktime_get+0x3a/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.689046] [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.692915] [<ffffffff814ebcf6>] io_schedule_timeout+0xa6/0x110
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.697311] [<ffffffff814ecd4b>] bit_wait_io+0x1b/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.701238] [<ffffffff814ec990>] __wait_on_bit+0x60/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.705287] [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.709137] [<ffffffff814eca32>] out_of_line_wait_on_bit+0x72/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.713755] [<ffffffff810acf20>] ? autoremove_wake_function+0x40/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.718642] [<ffffffffa08abe83>] dm_bufio_write_dirty_buffers+0xe3/0x190 [dm_bufio]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.724344] [<ffffffffa08bd0bc>] dm_bm_flush+0x1c/0x20 [dm_persistent_data]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.729461] [<ffffffffa08bffc3>] dm_tm_commit+0x33/0x40 [dm_persistent_data]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.735149] [<ffffffffa08d8d39>] __commit_transaction+0x359/0x360 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.740735] [<ffffffffa08d9f47>] dm_pool_commit_metadata+0x37/0x60 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.746227] [<ffffffffa08d3967>] commit+0x27/0x90 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.750505] [<ffffffffa08d3cf2>] pool_status+0x322/0x6b0 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.755227] [<ffffffff811cd5c6>] ? __memcg_kmem_charge+0x46/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.759749] [<ffffffffa0889c96>] retrieve_status+0xa6/0x1b0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.764474] [<ffffffffa088ae14>] table_status+0x54/0x80 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.768939] [<ffffffffa088adc0>] ? dm_get_live_or_inactive_table.isra.3+0x30/0x30 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.775238] [<ffffffffa088aae6>] ctl_ioctl+0x246/0x4b0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.778569] [<ffffffff8107932b>] ? recalc_sigpending+0x1b/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.781825] [<ffffffffa088ad63>] dm_ctl_ioctl+0x13/0x20 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.785157] [<ffffffff811e772d>] do_vfs_ioctl+0x2cd/0x4b0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.788226] [<ffffffff8107c27b>] ? __set_current_blocked+0x4b/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.791642] [<ffffffff811e7989>] SyS_ioctl+0x79/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.794518] [<ffffffff8107c4a7>] ? SyS_rt_sigprocmask+0x67/0xb0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.797936] [<ffffffff814eff6e>] entry_SYSCALL_64_fastpath+0x12/0x71
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.801516] INFO: task rs:main Q:Reg:3401 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.805308] Tainted: G E 4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.808942] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.813403] rs:main Q:Reg D ffff881e6a3bf968 0 3401 1 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.817403] ffff881e6a3bf968 ffff881e77ab0000 ffff881e71ddba80 ffff881e6a3c0000
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.822043] 0000000000000000 7fffffffffffffff ffff881efff92a08 ffffffff814ecd30
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.840036] ffff881e6a3bf980 ffffffff814ec5f5 ffff881e84695340 ffff881e6a3bfa20
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.844684] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.846326] [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.850013] [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.853482] [<ffffffff814eef01>] schedule_timeout+0x231/0x2b0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.857646] [<ffffffff8142c315>] ? netif_rx_internal+0x35/0xf0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.861887] [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.866587] [<ffffffff810d2f7a>] ? ktime_get+0x3a/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.870317] [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.874015] [<ffffffff814ebcf6>] io_schedule_timeout+0xa6/0x110
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.878223] [<ffffffff814ecd4b>] bit_wait_io+0x1b/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.881947] [<ffffffff814ec990>] __wait_on_bit+0x60/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.885801] [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.889444] [<ffffffff814eca32>] out_of_line_wait_on_bit+0x72/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.893829] [<ffffffff810acf20>] ? autoremove_wake_function+0x40/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.898380] [<ffffffffa00ccdb9>] do_get_write_access+0x279/0x4c0 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.903081] [<ffffffff8120918e>] ? __find_get_block+0xae/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.907299] [<ffffffffa00cd031>] jbd2_journal_get_write_access+0x31/0x60 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.912781] [<ffffffffa0136511>] __ext4_journal_get_write_access+0x31/0x70 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.918323] [<ffffffffa01080da>] ext4_reserve_inode_write+0x6a/0x90 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.923190] [<ffffffffa010b888>] ? ext4_dirty_inode+0x48/0x70 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.927728] [<ffffffffa010813f>] ext4_mark_inode_dirty+0x3f/0x1f0 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.932491] [<ffffffffa010b888>] ext4_dirty_inode+0x48/0x70 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.936911] [<ffffffff812009e1>] __mark_inode_dirty+0x181/0x370
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.941163] [<ffffffff811ee887>] generic_update_time+0x87/0xd0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.945363] [<ffffffff811edfd9>] file_update_time+0x99/0xf0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.949398] [<ffffffff810dc784>] ? get_futex_key+0x194/0x270
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.953487] [<ffffffff81163f1c>] __generic_file_write_iter+0x9c/0x1e0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.958070] [<ffffffff810dcd41>] ? futex_wake+0x81/0x150
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.961918] [<ffffffffa00feb76>] ext4_file_write_iter+0xf6/0x360 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.966561] [<ffffffff810dec50>] ? do_futex+0x100/0xad0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.970384] [<ffffffff81411c68>] ? sock_sendmsg+0x38/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.974311] [<ffffffff814120e9>] ? SYSC_sendto+0x139/0x170
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.978286] [<ffffffff811d4dba>] __vfs_write+0xaa/0xe0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.982034] [<ffffffff811d53e2>] vfs_write+0xa2/0x1a0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.985716] [<ffffffff811d60e6>] SyS_write+0x46/0xa0
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.989368] [<ffffffff814eff6e>] entry_SYSCALL_64_fastpath+0x12/0x71
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.993848] INFO: task agent:43861 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365557.998607] Tainted: G E 4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.003736] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.009987] agent D ffff880c4fdf7860 0 43861 13580 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.015615] ffff880c4fdf7860 ffff881e77ab0000 ffff881d07de57c0 ffff880c4fdf8000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.022129] ffff881e70ff6558 ffff881e70ff6570 ffff880c4fdf7990 0000000000000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.028637] ffff880c4fdf7878 ffffffff814ec5f5 ffff881d07de57c0 ffff880c4fdf78c8
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.035142] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.037302] [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.041067] [<ffffffff814eec2a>] rwsem_down_read_failed+0xea/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.045637] [<ffffffff812dabd4>] call_rwsem_down_read_failed+0x14/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.050637] [<ffffffff814ee2d7>] ? down_read+0x17/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.054694] [<ffffffffa08d98ae>] dm_thin_find_block+0x3e/0xd0 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.060370] [<ffffffffa08d55cc>] thin_map+0x13c/0x280 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.065245] [<ffffffffa0881eca>] __map_bio+0x3a/0x110 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.069759] [<ffffffffa0883630>] __split_and_process_bio+0x240/0x3c0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.075500] [<ffffffffa088381a>] dm_make_request+0x6a/0xd0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.080323] [<ffffffff812aa0c4>] generic_make_request+0xe4/0x1a0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.084977] [<ffffffff812aa1e8>] submit_bio+0x68/0x130
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.089021] [<ffffffffa014d6e5>] ext4_mpage_readpages+0x2c5/0x7c0 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.094118] [<ffffffff811af068>] ? alloc_pages_current+0x88/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.098800] [<ffffffffa0103307>] ext4_readpages+0x37/0x40 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.103485] [<ffffffff8116e484>] __do_page_cache_readahead+0x174/0x200
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.108597] [<ffffffff81164714>] filemap_fault+0x334/0x3e0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.113007] [<ffffffffa010bed6>] ext4_filemap_fault+0x36/0x50 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.118027] [<ffffffff8118bfbd>] __do_fault+0x3d/0x70
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.122134] [<ffffffff811903f7>] handle_mm_fault+0xf27/0x1870
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.126725] [<ffffffff810b3501>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.133188] [<ffffffff8105ed33>] __do_page_fault+0x183/0x3f0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.137660] [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.142909] [<ffffffff8105efc2>] do_page_fault+0x22/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.147324] [<ffffffff814f2058>] page_fault+0x28/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.151830] INFO: task consul:15336 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.157299] Tainted: G E 4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.162777] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.169393] consul D ffff881e4f6a3860 0 15336 15293 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.175451] ffff881e4f6a3860 ffffffff81a11500 ffff881e49d2ba80 ffff881e4f6a4000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.182413] ffff881e70ff6558 ffff881e70ff6570 ffff881e4f6a3990 0000000000000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.189346] ffff881e4f6a3878 ffffffff814ec5f5 ffff881e49d2ba80 ffff881e4f6a38c8
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.196075] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.198259] [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.202177] [<ffffffff814eec2a>] rwsem_down_read_failed+0xea/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.206997] [<ffffffff812dabd4>] call_rwsem_down_read_failed+0x14/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.212061] [<ffffffff814ee2d7>] ? down_read+0x17/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.216155] [<ffffffffa08d98ae>] dm_thin_find_block+0x3e/0xd0 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.221855] [<ffffffffa08d55cc>] thin_map+0x13c/0x280 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.226781] [<ffffffffa0881eca>] __map_bio+0x3a/0x110 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.231363] [<ffffffffa0883630>] __split_and_process_bio+0x240/0x3c0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.237145] [<ffffffffa088381a>] dm_make_request+0x6a/0xd0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.241819] [<ffffffff812aa0c4>] generic_make_request+0xe4/0x1a0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.246294] [<ffffffff812aa1e8>] submit_bio+0x68/0x130
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.250354] [<ffffffff8117028e>] ? lru_cache_add+0xe/0x10
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.254594] [<ffffffff81162c1d>] ? add_to_page_cache_lru+0x4d/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.259375] [<ffffffffa014d6e5>] ext4_mpage_readpages+0x2c5/0x7c0 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.264526] [<ffffffff811af068>] ? alloc_pages_current+0x88/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.269243] [<ffffffffa0103307>] ext4_readpages+0x37/0x40 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.273910] [<ffffffff8116e484>] __do_page_cache_readahead+0x174/0x200
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.278920] [<ffffffff81164714>] filemap_fault+0x334/0x3e0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.283247] [<ffffffffa010bed6>] ext4_filemap_fault+0x36/0x50 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.288152] [<ffffffff8118bfbd>] __do_fault+0x3d/0x70
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.292114] [<ffffffff811903f7>] handle_mm_fault+0xf27/0x1870
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.296529] [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.301583] [<ffffffff8105ed33>] __do_page_fault+0x183/0x3f0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.306036] [<ffffffff8105efc2>] do_page_fault+0x22/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.310156] [<ffffffff814f2058>] page_fault+0x28/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.314101] INFO: task consul:15339 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.318987] Tainted: G E 4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.324015] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.330096] consul D ffff881e4f69bd98 0 15339 15293 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.335620] ffff881e4f69bd98 ffff881e77a9ba80 ffff881e51918000 ffff881e4f69c000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.341683] ffff881e749dbc80 ffff881e749dbc68 ffffffff00000000 ffffffff00000002
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.347735] ffff881e4f69bdb0 ffffffff814ec5f5 ffff881e51918000 ffff881e4f69be30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.354237] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.356452] [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.360397] [<ffffffff814ee9fa>] rwsem_down_write_failed+0x1da/0x320
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.365314] [<ffffffff810a03b5>] ? pick_next_entity+0xa5/0x160
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.369767] [<ffffffff812dac03>] call_rwsem_down_write_failed+0x13/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.374620] [<ffffffff814ee304>] ? down_write+0x24/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.378673] [<ffffffff8117d53b>] vm_mmap_pgoff+0x6b/0xc0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.382693] [<ffffffff811955b3>] SyS_mmap_pgoff+0x103/0x260
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.386914] [<ffffffff810d240a>] ? ktime_get_ts64+0x4a/0xe0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.391291] [<ffffffff8101b292>] SyS_mmap+0x22/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.395186] [<ffffffff814eff6e>] entry_SYSCALL_64_fastpath+0x12/0x71
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.400120] INFO: task consul:15340 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.405105] Tainted: G E 4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.410223] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.416474] consul D ffff881e4f657860 0 15340 15293 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.422156] ffff881e4f657860 ffff881e77aeba80 ffff881e69c9ba80 ffff881e4f658000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.428666] ffff881e70ff6558 ffff881e70ff6570 ffff881e4f657990 0000000000000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.435155] ffff881e4f657878 ffffffff814ec5f5 ffff881e69c9ba80 ffff881e4f6578c8
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.441440] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.443518] [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.447315] [<ffffffff814eec2a>] rwsem_down_read_failed+0xea/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.452132] [<ffffffff812dabd4>] call_rwsem_down_read_failed+0x14/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.457138] [<ffffffff814ee2d7>] ? down_read+0x17/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.461221] [<ffffffffa08d98ae>] dm_thin_find_block+0x3e/0xd0 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.466932] [<ffffffffa08d55cc>] thin_map+0x13c/0x280 [dm_thin_pool]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.471825] [<ffffffffa0881eca>] __map_bio+0x3a/0x110 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.476386] [<ffffffffa0883630>] __split_and_process_bio+0x240/0x3c0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.482138] [<ffffffffa088381a>] dm_make_request+0x6a/0xd0 [dm_mod]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.486966] [<ffffffff812aa0c4>] generic_make_request+0xe4/0x1a0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.491617] [<ffffffff812aa1e8>] submit_bio+0x68/0x130
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.495686] [<ffffffff8117028e>] ? lru_cache_add+0xe/0x10
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.499933] [<ffffffff81162c1d>] ? add_to_page_cache_lru+0x4d/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.504756] [<ffffffffa014d6e5>] ext4_mpage_readpages+0x2c5/0x7c0 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.509903] [<ffffffff811af068>] ? alloc_pages_current+0x88/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.514668] [<ffffffffa0103307>] ext4_readpages+0x37/0x40 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.519387] [<ffffffff8116e484>] __do_page_cache_readahead+0x174/0x200
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.524455] [<ffffffff81164714>] filemap_fault+0x334/0x3e0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.528782] [<ffffffffa010bed6>] ext4_filemap_fault+0x36/0x50 [ext4]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.533708] [<ffffffff8118bfbd>] __do_fault+0x3d/0x70
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.537682] [<ffffffff811903f7>] handle_mm_fault+0xf27/0x1870
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.541997] [<ffffffff8147670a>] ? tcp_poll+0x6a/0x240
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.545909] [<ffffffff810b3501>] ? __raw_callee_save___pv_queued_spin_unlock+0x11/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.552179] [<ffffffff8121a631>] ? SyS_epoll_ctl+0xd1/0xca0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.556523] [<ffffffff8105ed33>] __do_page_fault+0x183/0x3f0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.574325] [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.579398] [<ffffffff8105efc2>] do_page_fault+0x22/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.583584] [<ffffffff814f2058>] page_fault+0x28/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.587630] INFO: task jbd2/dm-10-8:16273 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.593052] Tainted: G E 4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.598254] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.604597] jbd2/dm-10-8 D ffff881df5e17b08 0 16273 2 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.610381] ffff881df5e17b08 ffffffff81a11500 ffff881e51bcba80 ffff881df5e18000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.617026] 0000000000000000 7fffffffffffffff ffff881efff94478 ffffffff814ecd30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.623543] ffff881df5e17b20 ffffffff814ec5f5 ffff881e84615340 ffff881df5e17bb8
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.630097] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.632304] [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.636368] [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.640199] [<ffffffff814eef01>] schedule_timeout+0x231/0x2b0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.644499] [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.649548] [<ffffffff810d2f7a>] ? ktime_get+0x3a/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.653643] [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.657688] [<ffffffff814ebcf6>] io_schedule_timeout+0xa6/0x110
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.662327] [<ffffffff814ecd4b>] bit_wait_io+0x1b/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.666432] [<ffffffff814ec990>] __wait_on_bit+0x60/0x90
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.670655] [<ffffffff814ecd30>] ? bit_wait+0x50/0x50
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.674676] [<ffffffff814eca32>] out_of_line_wait_on_bit+0x72/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.679505] [<ffffffff810acf20>] ? autoremove_wake_function+0x40/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.684492] [<ffffffff812088c7>] __wait_on_buffer+0x27/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.688881] [<ffffffffa00cf2f6>] jbd2_journal_commit_transaction+0xf66/0x1bb0 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.695040] [<ffffffffa00d3c90>] kjournald2+0x120/0x340 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.699622] [<ffffffff810acee0>] ? prepare_to_wait_event+0xf0/0xf0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.704476] [<ffffffffa00d3b70>] ? commit_timeout+0x10/0x10 [jbd2]
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.709312] [<ffffffff8108a7f9>] kthread+0xc9/0xe0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.713179] [<ffffffff8108a730>] ? kthread_park+0x60/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.717631] [<ffffffff814f02cf>] ret_from_fork+0x3f/0x70
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.721865] [<ffffffff8108a730>] ? kthread_park+0x60/0x60
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.726156] INFO: task java:16402 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.731055] Tainted: G E 4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.736209] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.742194] java D ffff881e6bc37e08 0 16402 16351 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.747635] ffff881e6bc37e08 ffff881e77a99d40 ffff881e61288000 ffff881e6bc38000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.754131] ffff881e75cd9340 ffff881e75cd9328 ffffffff00000000 ffffffff00000001
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.760631] ffff881e6bc37e20 ffffffff814ec5f5 ffff881e61288000 ffff881e6bc37ea0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.767086] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.769269] [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.773139] [<ffffffff814ee9fa>] rwsem_down_write_failed+0x1da/0x320
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.778030] [<ffffffff812dac03>] call_rwsem_down_write_failed+0x13/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.783083] [<ffffffff814ee304>] ? down_write+0x24/0x40
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.787210] [<ffffffff811987d1>] SyS_mprotect+0xc1/0x240
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.791414] [<ffffffff810cefd1>] ? posix_ktime_get_ts+0x11/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.796010] [<ffffffff814eff6e>] entry_SYSCALL_64_fastpath+0x12/0x71
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.800940] INFO: task java:16445 blocked for more than 120 seconds.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.805799] Tainted: G E 4.4.51-40.58.amzn1.x86_64 #1
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.810897] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.817115] java D ffff881debbd3e10 0 16445 16351 0x00000000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.822756] ffff881debbd3e10 ffffffff81a11500 ffff881e6bd2ba80 ffff881debbd4000
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.829278] ffff881e75cd9328 ffff881e75cd9340 ffff881debbd3f58 ffff881e6bd2ba80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.835788] ffff881debbd3e28 ffffffff814ec5f5 ffff881e6bd2ba80 ffff881debbd3e78
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.842069] Call Trace:
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.844150] [<ffffffff814ec5f5>] schedule+0x35/0x80
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.847945] [<ffffffff814eec2a>] rwsem_down_read_failed+0xea/0x120
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.852755] [<ffffffff812dabd4>] call_rwsem_down_read_failed+0x14/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.857772] [<ffffffff814ee2d7>] ? down_read+0x17/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.861861] [<ffffffff8105ef39>] __do_page_fault+0x389/0x3f0
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.866301] [<ffffffff8100c365>] ? xen_clocksource_get_cycles+0x15/0x20
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.871390] [<ffffffff8105efc2>] do_page_fault+0x22/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365558.875588] [<ffffffff814f2058>] page_fault+0x28/0x30
May 17 06:50:19 ip-172-16-15-126 kernel: [365665.548153] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 858.
May 17 06:50:19 ip-172-16-15-126 kernel: [365703.557562] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 229.
May 17 06:50:19 ip-172-16-15-126 kernel: [365704.561666] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 567.
May 17 06:50:19 ip-172-16-15-126 kernel: [365704.566122] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 1021.
May 17 06:50:19 ip-172-16-15-126 kernel: [365705.569628] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 647.
May 17 06:50:19 ip-172-16-15-126 kernel: [365705.573997] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 648.
May 17 06:50:19 ip-172-16-15-126 kernel: [365706.577713] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 230.
May 17 06:50:19 ip-172-16-15-126 kernel: [365707.581763] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 876.
May 17 06:50:19 ip-172-16-15-126 kernel: [365708.585837] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 649.
May 17 06:50:19 ip-172-16-15-126 kernel: [365709.593807] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 809.
May 17 06:50:19 ip-172-16-15-126 kernel: [365709.598182] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 231.
May 17 06:50:19 ip-172-16-15-126 kernel: [365711.601901] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 650.
May 17 06:50:19 ip-172-16-15-126 kernel: [365712.605951] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 810.
May 17 06:50:19 ip-172-16-15-126 kernel: [365714.610007] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 651.
May 17 06:50:19 ip-172-16-15-126 kernel: [365724.614383] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 232.
May 17 06:50:19 ip-172-16-15-126 kernel: [365730.618610] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 0, index 87.
May 17 06:50:19 ip-172-16-15-126 kernel: [365737.622898] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 872.
May 17 06:50:19 ip-172-16-15-126 kernel: [365739.626961] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 839.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.631063] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 0, index 88.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.635070] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 0, index 89.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.639040] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 840.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.643015] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 841.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.647046] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 811.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.651030] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 812.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.655302] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 813.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.659545] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 814.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.663826] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 815.
May 17 06:50:19 ip-172-16-15-126 kernel: [365742.668055] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 233.
May 17 06:50:19 ip-172-16-15-126 kernel: [365743.671120] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 1022.
May 17 06:50:19 ip-172-16-15-126 kernel: [365743.675192] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 1023.
May 17 06:50:19 ip-172-16-15-126 kernel: [365744.683098] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 652.
May 17 06:50:19 ip-172-16-15-126 kernel: [365744.687117] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 653.
May 17 06:50:19 ip-172-16-15-126 kernel: [365746.691242] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 0.
May 17 06:50:19 ip-172-16-15-126 kernel: [365747.695279] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 654.
May 17 06:50:19 ip-172-16-15-126 kernel: [365748.699313] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 842.
May 17 06:50:19 ip-172-16-15-126 kernel: [365749.703345] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 568.
May 17 06:50:19 ip-172-16-15-126 kernel: [365749.707929] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 569.
May 17 06:50:19 ip-172-16-15-126 kernel: [365749.712383] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 1.
May 17 06:50:19 ip-172-16-15-126 kernel: [365749.716774] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 2.
May 17 06:50:19 ip-172-16-15-126 kernel: [365763.723946] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 816.
May 17 06:50:19 ip-172-16-15-126 kernel: [365894.752829] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 899.
May 17 06:50:19 ip-172-16-15-126 kernel: [366004.772985] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 14.
May 17 06:50:19 ip-172-16-15-126 kernel: [366006.777073] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 849.
May 17 06:50:19 ip-172-16-15-126 kernel: [366007.781113] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 15.
May 17 06:50:19 ip-172-16-15-126 kernel: [366007.785109] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 911.
May 17 06:50:19 ip-172-16-15-126 kernel: [366012.789315] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 850.
May 17 06:50:19 ip-172-16-15-126 kernel: [366013.793349] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 16.
May 17 06:50:19 ip-172-16-15-126 kernel: [366025.797802] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 919.
May 17 06:50:19 ip-172-16-15-126 kernel: [366027.801877] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 248.
May 17 06:50:19 ip-172-16-15-126 kernel: [366030.805996] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 1, index 851.
May 17 06:50:19 ip-172-16-15-126 kernel: [366031.810023] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 586.
May 17 06:50:19 ip-172-16-15-126 kernel: [366031.814046] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 587.
May 17 06:50:19 ip-172-16-15-126 kernel: [366044.830516] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 669.
May 17 06:50:19 ip-172-16-15-126 kernel: [366060.835104] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 0, index 102.
May 17 06:50:19 ip-172-16-15-126 kernel: [366060.839915] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 822.
May 17 06:50:19 ip-172-16-15-126 kernel: [366061.843146] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 588.
May 17 06:50:19 ip-172-16-15-126 kernel: [366063.847294] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 3, index 249.
May 17 06:50:19 ip-172-16-15-126 kernel: [366072.863541] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 823.
May 17 06:50:19 ip-172-16-15-126 kernel: [366074.863683] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 924.
May 17 06:50:19 ip-172-16-15-126 kernel: [366100.868637] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 589.
May 17 06:50:19 ip-172-16-15-126 kernel: [366103.872750] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 17.
May 17 06:50:19 ip-172-16-15-126 kernel: [366103.876745] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 18.
May 17 06:50:19 ip-172-16-15-126 kernel: [366105.880828] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 824.
May 17 06:50:19 ip-172-16-15-126 kernel: [366105.884859] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 825.
May 17 06:50:19 ip-172-16-15-126 kernel: [366109.888993] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 5, index 23.
May 17 06:50:19 ip-172-16-15-126 kernel: [366117.893301] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 2, index 827.
May 17 06:50:19 ip-172-16-15-126 kernel: [366283.903580] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 6, index 962.
May 17 06:50:19 ip-172-16-15-126 kernel: [366305.908408] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 682.
May 17 06:50:19 ip-172-16-15-126 kernel: [366313.912713] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 596.
May 17 06:50:19 ip-172-16-15-126 kernel: [366313.917037] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 4, index 597.
May 17 06:50:19 ip-172-16-15-126 kernel: [366314.920749] ena 0000:00:03.0 eth0: Found a Tx that wasn't completed on time, qid 7, index 683.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment