forked from projecteru/redis-cerberus
-
Notifications
You must be signed in to change notification settings - Fork 0
v0.7.10版本rsp_to方法 core dump检验日志
Lewis Chan edited this page Mar 27, 2017
·
1 revision
当客户端以很小的超时时间发送命令时，cerberus很容易产生segmentation fault错误，具体backstrace结果：
(gdb) bt
#0 0x0000003a47432625 in raise () from /lib64/libc.so.6
#1 0x0000003a47433e05 in abort () from /lib64/libc.so.6
#2 0x0000003a4a46007d in __gnu_cxx::__verbose_terminate_handler () at ../../.././libstdc++-v3/libsupc++/vterminate.cc:95
#3 0x0000003a4a45e0e6 in __cxxabiv1::__terminate (handler=<optimized out>) at ../../.././libstdc++-v3/libsupc++/eh_terminate.cc:47
#4 0x0000003a4a45e131 in std::terminate () at ../../.././libstdc++-v3/libsupc++/eh_terminate.cc:57
#5 0x000000000053dafc in handle_segv(int, siginfo*, void*) ()
#6 <signal handler called>
#7 0x00002ae888003400 in ?? ()
#8 0x000000000051f423 in (anonymous namespace)::NormalResponse::rsp_to (this=0x2ae88803b470, cmd=...) at core/response.cpp:30
#9 0x0000000000522d6a in cerb::Server::_recv_from (this=0x2ae8880c3d50) at core/server.cpp:86
#10 0x0000000000521b47 in cerb::Server::on_events (this=0x2ae8880c3d50, events=1) at core/server.cpp:27
#11 0x0000000000513371 in cerb::Proxy::handle_events (this=0x2ae884b05010, events=0x2ae884f65da0, nfds=4) at core/proxy.cpp:298
#12 0x0000000000506ba2 in operator() (this=0x119ec28) at core/concurrence.cpp:24
#13 0x0000000000506ae5 in std::_Bind_simple<cerb::ListenThread::run()::$_0 ()>::_M_invoke<>(std::_Index_tuple<>) (this=0x119ec28)
at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.3/../../../../include/c++/4.9.3/functional:1699
#14 0x0000000000506ab5 in std::_Bind_simple<cerb::ListenThread::run()::$_0 ()>::operator()() (this=0x119ec28)
at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.3/../../../../include/c++/4.9.3/functional:1688
#15 0x0000000000506a8c in std::thread::_Impl<std::_Bind_simple<cerb::ListenThread::run()::$_0 ()> >::_M_run() (this=0x119ec10)
at /usr/bin/../lib/gcc/x86_64-unknown-linux-gnu/4.9.3/../../../../include/c++/4.9.3/thread:115
#16 0x0000003a4a4bb760 in std::(anonymous namespace)::execute_native_thread_routine (__p=<optimized out>)
at ../../../.././libstdc++-v3/src/c++11/thread.cc:84
#17 0x0000003a47807aa1 in start_thread () from /lib64/libpthread.so.0
#18 0x0000003a474e893d in clone () from /lib64/libc.so.6
bt显示在rsp_to处发生未知错误。猜测可能是cmd参数失效。经过在源码相关地方添加日志，得出某一次运行问题线程的输出：
2017-03-27 10:05:21,660 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,660 D 47178151532288 *poll Acceptor(7@0x2ae884b250f8)
2017-03-27 10:05:21,660 D 47178151532288 ACCEPT CLIENT fd=12
2017-03-27 10:05:21,660 D 47178151532288 Should update slot map
2017-03-27 10:05:21,661 D 47178151532288 Create SlotsMapUpdater(13@0x2ae888000dc0)[10.221.48.214:19308]
2017-03-27 10:05:21,662 D 47178151532288 *poll done
2017-03-27 10:05:21,680 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,680 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,680 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,681 D 47178151532288 -Keyslot = 15217
2017-03-27 10:05:21,681 W 47178151532288 cmd@0x2ae888000f70 created
2017-03-27 10:05:21,681 D 47178151532288 Client(12@0x2ae888000aa0) Process 1 over 1 commands
2017-03-27 10:05:21,681 D 47178151532288 Cluster slot not covered 15217
2017-03-27 10:05:21,681 D 47178151532288 Retry later: 0x2ae888000f70 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,681 W 47178151532288 in _process() peer,size=0
2017-03-27 10:05:21,681 D 47178151532288 Processed, rest buffer 0
2017-03-27 10:05:21,681 D 47178151532288 *poll done
2017-03-27 10:05:21,685 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,686 D 47178151532288 *poll SlotsMapUpdater(13@0x2ae888000dc0)[10.221.48.214:19308]
2017-03-27 10:05:21,686 D 47178151532288 *poll done
2017-03-27 10:05:21,690 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,690 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,690 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,690 D 47178151532288 -Keyslot = 11088
2017-03-27 10:05:21,690 W 47178151532288 cmd@0x2ae888000ed0 created
2017-03-27 10:05:21,690 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,690 D 47178151532288 *poll done
2017-03-27 10:05:21,693 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,694 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,694 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,696 D 47178151532288 -Keyslot = 6847
2017-03-27 10:05:21,696 W 47178151532288 cmd@0x2ae888001260 created
2017-03-27 10:05:21,696 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,696 D 47178151532288 *poll done
2017-03-27 10:05:21,698 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,698 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,700 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,700 D 47178151532288 -Keyslot = 2718
2017-03-27 10:05:21,700 W 47178151532288 cmd@0x2ae888000fb0 created
2017-03-27 10:05:21,700 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,700 D 47178151532288 *poll done
2017-03-27 10:05:21,704 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,704 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,704 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,704 D 47178151532288 -Keyslot = 15101
2017-03-27 10:05:21,704 W 47178151532288 cmd@0x2ae888001110 created
2017-03-27 10:05:21,704 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,704 D 47178151532288 *poll done
2017-03-27 10:05:21,707 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,707 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,707 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,708 D 47178151532288 -Keyslot = 10972
2017-03-27 10:05:21,708 W 47178151532288 cmd@0x2ae8880013a0 created
2017-03-27 10:05:21,709 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,709 D 47178151532288 *poll done
2017-03-27 10:05:21,710 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,710 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,710 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,710 D 47178151532288 -Keyslot = 6715
2017-03-27 10:05:21,710 W 47178151532288 cmd@0x2ae888001530 created
2017-03-27 10:05:21,710 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,710 D 47178151532288 *poll done
2017-03-27 10:05:21,711 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,711 D 47178151532288 *poll SlotsMapUpdater(13@0x2ae888000dc0)[10.221.48.214:19308]
2017-03-27 10:05:21,711 D 47178151532288 *Updated from SlotsMapUpdater(13@0x2ae888000dc0)[10.221.48.214:19308]
2017-03-27 10:05:21,775 D 47178151532288 CLOSE fd=13
2017-03-27 10:05:21,776 D 47178151532288 Allocate Server: 0x2ae8880c3240
2017-03-27 10:05:21,776 D 47178151532288 Allocate Server: 0x2ae8880c3690
2017-03-27 10:05:21,776 D 47178151532288 Allocate Server: 0x2ae8880c39f0
2017-03-27 10:05:21,776 D 47178151532288 Allocate Server: 0x2ae8880c3d50
2017-03-27 10:05:21,776 D 47178151532288 Allocate Server: 0x2ae8880c40b0
2017-03-27 10:05:21,776 D 47178151532288 Allocate Server: 0x2ae8880c4460
2017-03-27 10:05:21,776 D 47178151532288 Allocate Server: 0x2ae8880c47c0
2017-03-27 10:05:21,777 D 47178151532288 Allocate Server: 0x2ae8880c4b20
2017-03-27 10:05:21,777 I 47178151532288 Open Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,778 D 47178151532288 Get Server(13@0x2ae8880c4b20) for 10.221.48.212:19308
2017-03-27 10:05:21,780 I 47178151532288 Open Server(26@0x2ae8880c47c0)
2017-03-27 10:05:21,780 D 47178151532288 Get Server(26@0x2ae8880c47c0) for 10.221.48.50:19308
2017-03-27 10:05:21,780 I 47178151532288 Open Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,780 D 47178151532288 Get Server(27@0x2ae8880c4460) for 10.221.48.251:19308
2017-03-27 10:05:21,785 I 47178151532288 Open Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,785 D 47178151532288 Get Server(28@0x2ae8880c40b0) for 10.221.48.250:19308
2017-03-27 10:05:21,790 I 47178151532288 Open Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,790 D 47178151532288 Get Server(20@0x2ae8880c3d50) for 10.221.48.215:19308
2017-03-27 10:05:21,793 I 47178151532288 Slot map updated
2017-03-27 10:05:21,793 D 47178151532288 Retry MOVED or ASK: 1
2017-03-27 10:05:21,794 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae888000f70
2017-03-27 10:05:21,794 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,799 I 47178151532288 Poll elapse=0.0882725 events=1 clients=1 long_clients=0 slots_map_updated=true
2017-03-27 10:05:21,800 D 47178151532288 *poll done
2017-03-27 10:05:21,800 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,800 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,800 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 312 read returns 312
2017-03-27 10:05:21,802 D 47178151532288 -Keyslot = 2586
2017-03-27 10:05:21,802 W 47178151532288 cmd@0x2ae8880039b0 created
2017-03-27 10:05:21,802 D 47178151532288 -Keyslot = 14969
2017-03-27 10:05:21,802 W 47178151532288 cmd@0x2ae888003480 created
2017-03-27 10:05:21,802 D 47178151532288 -Keyslot = 10840
2017-03-27 10:05:21,802 W 47178151532288 cmd@0x2ae8880014c0 created
2017-03-27 10:05:21,803 D 47178151532288 -Keyslot = 2112
2017-03-27 10:05:21,803 W 47178151532288 cmd@0x2ae888003410 created
2017-03-27 10:05:21,803 D 47178151532288 -Keyslot = 6241
2017-03-27 10:05:21,803 W 47178151532288 cmd@0x2ae888003140 created
2017-03-27 10:05:21,803 D 47178151532288 -Keyslot = 10638
2017-03-27 10:05:21,803 W 47178151532288 cmd@0x2ae8880384f0 created
2017-03-27 10:05:21,803 D 47178151532288 -Keyslot = 14767
2017-03-27 10:05:21,803 W 47178151532288 cmd@0x2ae88803d230 created
2017-03-27 10:05:21,803 D 47178151532288 -Keyslot = 2508
2017-03-27 10:05:21,803 W 47178151532288 cmd@0x2ae88803d1f0 created
2017-03-27 10:05:21,803 D 47178151532288 -Keyslot = 6637
2017-03-27 10:05:21,803 W 47178151532288 cmd@0x2ae8880c3090 created
2017-03-27 10:05:21,803 D 47178151532288 -Keyslot = 10506
2017-03-27 10:05:21,803 W 47178151532288 cmd@0x2ae888097260 created
2017-03-27 10:05:21,803 D 47178151532288 -Keyslot = 14635
2017-03-27 10:05:21,803 W 47178151532288 cmd@0x2ae88806b4b0 created
2017-03-27 10:05:21,803 D 47178151532288 -Keyslot = 2376
2017-03-27 10:05:21,804 W 47178151532288 cmd@0x2ae8880972e0 created
2017-03-27 10:05:21,804 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,804 D 47178151532288 *poll done
2017-03-27 10:05:21,804 D 47178151532288 *poll nfds: 2
2017-03-27 10:05:21,805 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,808 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 52 read returns 52
2017-03-27 10:05:21,808 D 47178151532288 -Keyslot = 6505
2017-03-27 10:05:21,808 W 47178151532288 cmd@0x2ae8880972a0 created
2017-03-27 10:05:21,809 D 47178151532288 -Keyslot = 6270
2017-03-27 10:05:21,809 W 47178151532288 cmd@0x2ae88803d1b0 created
2017-03-27 10:05:21,809 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,809 D 47178151532288 *poll Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,809 W 47178151532288 Server(13@0x2ae8880c4b20) push _commands to _sent_commands done
2017-03-27 10:05:21,809 D 47178151532288 *poll done
2017-03-27 10:05:21,809 D 47178151532288 *poll nfds: 4
2017-03-27 10:05:21,809 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,809 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,809 D 47178151532288 -Keyslot = 2143
2017-03-27 10:05:21,809 W 47178151532288 cmd@0x2ae88803d270 created
2017-03-27 10:05:21,809 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,809 D 47178151532288 *poll Server(26@0x2ae8880c47c0)
2017-03-27 10:05:21,809 W 47178151532288 Server(26@0x2ae8880c47c0) push _commands to _sent_commands done
2017-03-27 10:05:21,810 D 47178151532288 *poll Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,810 W 47178151532288 Server(27@0x2ae8880c4460) push _commands to _sent_commands done
2017-03-27 10:05:21,810 D 47178151532288 *poll Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,810 W 47178151532288 Server(28@0x2ae8880c40b0) push _commands to _sent_commands done
2017-03-27 10:05:21,810 D 47178151532288 Write to 28 : 26 bytes written
2017-03-27 10:05:21,810 D 47178151532288 *poll done
2017-03-27 10:05:21,813 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,813 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,813 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,813 D 47178151532288 -Keyslot = 14768
2017-03-27 10:05:21,813 W 47178151532288 cmd@0x2ae888002cd0 created
2017-03-27 10:05:21,813 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,813 D 47178151532288 *poll done
2017-03-27 10:05:21,814 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,814 D 47178151532288 *poll Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,814 W 47178151532288 Server(20@0x2ae8880c3d50) push _commands to _sent_commands done
2017-03-27 10:05:21,814 D 47178151532288 *poll done
2017-03-27 10:05:21,816 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,816 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,816 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,816 D 47178151532288 -Keyslot = 10641
2017-03-27 10:05:21,816 W 47178151532288 cmd@0x2ae8880c5290 created
2017-03-27 10:05:21,816 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,816 D 47178151532288 *poll done
2017-03-27 10:05:21,820 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,820 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,822 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,822 D 47178151532288 -Keyslot = 6642
2017-03-27 10:05:21,822 W 47178151532288 cmd@0x2ae8880c52d0 created
2017-03-27 10:05:21,822 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,822 D 47178151532288 *poll done
2017-03-27 10:05:21,833 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,833 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,833 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,833 D 47178151532288 -Keyslot = 2515
2017-03-27 10:05:21,833 W 47178151532288 cmd@0x2ae8880c51a0 created
2017-03-27 10:05:21,833 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,833 D 47178151532288 *poll done
2017-03-27 10:05:21,834 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,834 D 47178151532288 *poll Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,836 D 47178151532288 Read Server(28@0x2ae8880c40b0) buffer size 1041
2017-03-27 10:05:21,836 D 47178151532288 +responses size: 1
2017-03-27 10:05:21,836 D 47178151532288 +rest buffer: 0
2017-03-27 10:05:21,836 W 47178151532288 before rsp_to, cmd@0x2ae888000f70 in Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,836 W 47178151532288 Server(28@0x2ae8880c40b0) push _commands to _sent_commands done
2017-03-27 10:05:21,836 D 47178151532288 *poll done
2017-03-27 10:05:21,836 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,836 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,836 D 47178151532288 Write to 12 : 1041 bytes written
2017-03-27 10:05:21,836 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888000f70 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,836 W 47178151532288 before clear peer,size=1
2017-03-27 10:05:21,836 D 47178151532288 Client(12@0x2ae888000aa0) Process 25 over 25 commands
2017-03-27 10:05:21,836 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae888000ed0
2017-03-27 10:05:21,836 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,836 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae888001260
2017-03-27 10:05:21,836 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,836 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae888000fb0
2017-03-27 10:05:21,836 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,837 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae888001110
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,837 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae8880013a0
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,837 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae888001530
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,837 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae8880039b0
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,837 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae888003480
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,837 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae8880014c0
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,837 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae888003410
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,837 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae888003140
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,837 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae8880384f0
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,837 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae88803d230
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,837 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae88803d1f0
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,837 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae8880c3090
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,837 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae888097260
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,837 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae88806b4b0
2017-03-27 10:05:21,837 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,838 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae8880972e0
2017-03-27 10:05:21,838 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,838 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae8880972a0
2017-03-27 10:05:21,838 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,838 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae88803d1b0
2017-03-27 10:05:21,838 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,838 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae88803d270
2017-03-27 10:05:21,838 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,838 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae888002cd0
2017-03-27 10:05:21,838 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,838 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae8880c5290
2017-03-27 10:05:21,838 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,838 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae8880c52d0
2017-03-27 10:05:21,838 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,838 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae8880c51a0
2017-03-27 10:05:21,838 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,838 W 47178151532288 in _process() peer,size=4
2017-03-27 10:05:21,838 D 47178151532288 Processed, rest buffer 0
2017-03-27 10:05:21,838 D 47178151532288 *poll done
2017-03-27 10:05:21,838 D 47178151532288 *poll nfds: 5
2017-03-27 10:05:21,838 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,838 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,838 D 47178151532288 -Keyslot = 14644
2017-03-27 10:05:21,838 W 47178151532288 cmd@0x2ae8880c51e0 created
2017-03-27 10:05:21,838 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,839 D 47178151532288 *poll Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,839 W 47178151532288 Server(20@0x2ae8880c3d50) push _commands to _sent_commands done
2017-03-27 10:05:21,839 D 47178151532288 *writev to 20 iovcnt=7 total bytes=182
2017-03-27 10:05:21,839 D 47178151532288 *poll Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,839 W 47178151532288 Server(28@0x2ae8880c40b0) push _commands to _sent_commands done
2017-03-27 10:05:21,839 D 47178151532288 *writev to 28 iovcnt=5 total bytes=130
2017-03-27 10:05:21,839 D 47178151532288 *poll Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,839 W 47178151532288 Server(27@0x2ae8880c4460) push _commands to _sent_commands done
2017-03-27 10:05:21,839 D 47178151532288 *writev to 27 iovcnt=6 total bytes=156
2017-03-27 10:05:21,839 D 47178151532288 *poll Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,839 W 47178151532288 Server(13@0x2ae8880c4b20) push _commands to _sent_commands done
2017-03-27 10:05:21,839 D 47178151532288 *writev to 13 iovcnt=7 total bytes=182
2017-03-27 10:05:21,839 D 47178151532288 *poll done
2017-03-27 10:05:21,846 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,846 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,848 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,848 D 47178151532288 -Keyslot = 10517
2017-03-27 10:05:21,848 W 47178151532288 cmd@0x2ae888000f70 created
2017-03-27 10:05:21,849 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,849 D 47178151532288 *poll done
2017-03-27 10:05:21,857 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,857 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,857 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,857 D 47178151532288 -Keyslot = 6518
2017-03-27 10:05:21,857 W 47178151532288 cmd@0x2ae888002f40 created
2017-03-27 10:05:21,857 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,857 D 47178151532288 *poll done
2017-03-27 10:05:21,864 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,864 D 47178151532288 *poll Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,864 D 47178151532288 Read Server(20@0x2ae8880c3d50) buffer size 7287
2017-03-27 10:05:21,865 D 47178151532288 +responses size: 7
2017-03-27 10:05:21,865 D 47178151532288 +rest buffer: 0
2017-03-27 10:05:21,865 W 47178151532288 before rsp_to, cmd@0x2ae888000fb0 in Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,865 W 47178151532288 before rsp_to, cmd@0x2ae8880039b0 in Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,866 W 47178151532288 before rsp_to, cmd@0x2ae888003410 in Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,866 W 47178151532288 before rsp_to, cmd@0x2ae88803d1f0 in Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,866 W 47178151532288 before rsp_to, cmd@0x2ae8880972e0 in Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,866 W 47178151532288 before rsp_to, cmd@0x2ae88803d270 in Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,866 W 47178151532288 before rsp_to, cmd@0x2ae8880c51a0 in Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,866 W 47178151532288 Server(20@0x2ae8880c3d50) push _commands to _sent_commands done
2017-03-27 10:05:21,866 D 47178151532288 *poll done
2017-03-27 10:05:21,866 D 47178151532288 *poll nfds: 4
2017-03-27 10:05:21,866 D 47178151532288 *poll Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,867 D 47178151532288 Read Server(28@0x2ae8880c40b0) buffer size 5205
2017-03-27 10:05:21,867 D 47178151532288 +responses size: 5
2017-03-27 10:05:21,867 D 47178151532288 +rest buffer: 0
2017-03-27 10:05:21,867 W 47178151532288 before rsp_to, cmd@0x2ae888001110 in Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,867 W 47178151532288 before rsp_to, cmd@0x2ae888003480 in Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,867 W 47178151532288 before rsp_to, cmd@0x2ae88803d230 in Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,868 W 47178151532288 before rsp_to, cmd@0x2ae88806b4b0 in Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,868 W 47178151532288 before rsp_to, cmd@0x2ae888002cd0 in Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,868 W 47178151532288 Server(28@0x2ae8880c40b0) push _commands to _sent_commands done
2017-03-27 10:05:21,868 D 47178151532288 *poll Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,868 D 47178151532288 Read Server(27@0x2ae8880c4460) buffer size 5210
2017-03-27 10:05:21,869 D 47178151532288 +responses size: 6
2017-03-27 10:05:21,869 D 47178151532288 +rest buffer: 0
2017-03-27 10:05:21,869 W 47178151532288 before rsp_to, cmd@0x2ae888000ed0 in Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,869 W 47178151532288 before rsp_to, cmd@0x2ae8880013a0 in Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,869 W 47178151532288 before rsp_to, cmd@0x2ae8880014c0 in Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,869 W 47178151532288 before rsp_to, cmd@0x2ae8880384f0 in Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,869 W 47178151532288 before rsp_to, cmd@0x2ae888097260 in Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,869 W 47178151532288 before rsp_to, cmd@0x2ae8880c5290 in Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,869 W 47178151532288 Server(27@0x2ae8880c4460) push _commands to _sent_commands done
2017-03-27 10:05:21,876 D 47178151532288 *poll Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,880 D 47178151532288 Read Server(13@0x2ae8880c4b20) buffer size 7287
2017-03-27 10:05:21,881 D 47178151532288 +responses size: 7
2017-03-27 10:05:21,881 D 47178151532288 +rest buffer: 0
2017-03-27 10:05:21,881 W 47178151532288 before rsp_to, cmd@0x2ae888001260 in Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,881 W 47178151532288 before rsp_to, cmd@0x2ae888001530 in Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,881 W 47178151532288 before rsp_to, cmd@0x2ae888003140 in Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,881 W 47178151532288 before rsp_to, cmd@0x2ae8880c3090 in Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,881 W 47178151532288 before rsp_to, cmd@0x2ae8880972a0 in Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,881 W 47178151532288 before rsp_to, cmd@0x2ae88803d1b0 in Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,881 W 47178151532288 before rsp_to, cmd@0x2ae8880c52d0 in Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,885 W 47178151532288 Server(13@0x2ae8880c4b20) push _commands to _sent_commands done
2017-03-27 10:05:21,885 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,887 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 104 read returns 104
2017-03-27 10:05:21,892 D 47178151532288 -Keyslot = 2391
2017-03-27 10:05:21,892 W 47178151532288 cmd@0x2ae8880c3610 created
2017-03-27 10:05:21,896 D 47178151532288 -Keyslot = 11087
2017-03-27 10:05:21,896 W 47178151532288 cmd@0x2ae8880c5140 created
2017-03-27 10:05:21,896 D 47178151532288 -Keyslot = 15214
2017-03-27 10:05:21,896 W 47178151532288 cmd@0x2ae8880032b0 created
2017-03-27 10:05:21,896 D 47178151532288 -Keyslot = 2689
2017-03-27 10:05:21,896 W 47178151532288 cmd@0x2ae8880c3170 created
2017-03-27 10:05:21,896 D 47178151532288 Client(12@0x2ae888000aa0) Process 7 over 7 commands
2017-03-27 10:05:21,896 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae8880c51e0
2017-03-27 10:05:21,896 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,896 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae888000f70
2017-03-27 10:05:21,897 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,897 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae888002f40
2017-03-27 10:05:21,897 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,897 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae8880c3610
2017-03-27 10:05:21,897 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,897 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae8880c5140
2017-03-27 10:05:21,897 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,897 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae8880032b0
2017-03-27 10:05:21,897 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,897 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae8880c3170
2017-03-27 10:05:21,897 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,897 W 47178151532288 in _process() peer,size=4
2017-03-27 10:05:21,897 D 47178151532288 Processed, rest buffer 0
2017-03-27 10:05:21,897 D 47178151532288 *poll done
2017-03-27 10:05:21,897 D 47178151532288 *poll nfds: 5
2017-03-27 10:05:21,897 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,897 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,897 D 47178151532288 -Keyslot = 6816
2017-03-27 10:05:21,897 W 47178151532288 cmd@0x2ae888003cb0 created
2017-03-27 10:05:21,897 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,897 D 47178151532288 *writev to 12 iovcnt=25 total bytes=24989
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888000ed0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888001260 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888000fb0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888001110 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880013a0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888001530 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880039b0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888003480 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880014c0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888003410 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888003140 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880384f0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88803d230 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,898 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88803d1f0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,899 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c3090 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,899 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888097260 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,899 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88806b4b0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,899 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880972e0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,899 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880972a0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,899 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88803d1b0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,899 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88803d270 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,899 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888002cd0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,902 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c5290 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,902 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c52d0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,902 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c51a0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,902 W 47178151532288 before clear peer,size=4
2017-03-27 10:05:21,902 D 47178151532288 Client(12@0x2ae888000aa0) Process 1 over 1 commands
2017-03-27 10:05:21,902 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae888003cb0
2017-03-27 10:05:21,902 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,902 W 47178151532288 in _process() peer,size=1
2017-03-27 10:05:21,902 D 47178151532288 Processed, rest buffer 0
2017-03-27 10:05:21,902 D 47178151532288 *poll Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,902 W 47178151532288 Server(20@0x2ae8880c3d50) push _commands to _sent_commands done
2017-03-27 10:05:21,902 D 47178151532288 *writev to 20 iovcnt=2 total bytes=52
2017-03-27 10:05:21,902 D 47178151532288 *poll Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,902 W 47178151532288 Server(28@0x2ae8880c40b0) push _commands to _sent_commands done
2017-03-27 10:05:21,902 D 47178151532288 *writev to 28 iovcnt=2 total bytes=52
2017-03-27 10:05:21,903 D 47178151532288 *poll Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,903 W 47178151532288 Server(27@0x2ae8880c4460) push _commands to _sent_commands done
2017-03-27 10:05:21,903 D 47178151532288 *writev to 27 iovcnt=2 total bytes=52
2017-03-27 10:05:21,903 D 47178151532288 *poll Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,906 W 47178151532288 Server(13@0x2ae8880c4b20) push _commands to _sent_commands done
2017-03-27 10:05:21,906 D 47178151532288 *writev to 13 iovcnt=2 total bytes=52
2017-03-27 10:05:21,906 D 47178151532288 *poll done
2017-03-27 10:05:21,906 D 47178151532288 *poll nfds: 2
2017-03-27 10:05:21,906 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,909 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 650 read returns 650
2017-03-27 10:05:21,909 D 47178151532288 -Keyslot = 10947
2017-03-27 10:05:21,909 W 47178151532288 cmd@0x2ae8880c52d0 created
2017-03-27 10:05:21,910 D 47178151532288 -Keyslot = 15074
2017-03-27 10:05:21,910 W 47178151532288 cmd@0x2ae888003140 created
2017-03-27 10:05:21,910 D 47178151532288 -Keyslot = 2565
2017-03-27 10:05:21,910 W 47178151532288 cmd@0x2ae888003410 created
2017-03-27 10:05:21,910 D 47178151532288 -Keyslot = 6692
2017-03-27 10:05:21,910 W 47178151532288 cmd@0x2ae888000fb0 created
2017-03-27 10:05:21,910 D 47178151532288 -Keyslot = 10823
2017-03-27 10:05:21,910 W 47178151532288 cmd@0x2ae8880c51a0 created
2017-03-27 10:05:21,910 D 47178151532288 -Keyslot = 14950
2017-03-27 10:05:21,910 W 47178151532288 cmd@0x2ae8880c31d0 created
2017-03-27 10:05:21,910 D 47178151532288 -Keyslot = 15900
2017-03-27 10:05:21,910 W 47178151532288 cmd@0x2ae888000eb0 created
2017-03-27 10:05:21,910 D 47178151532288 -Keyslot = 11837
2017-03-27 10:05:21,910 W 47178151532288 cmd@0x2ae888003980 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 8146
2017-03-27 10:05:21,911 W 47178151532288 cmd@0x2ae888000e50 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 4083
2017-03-27 10:05:21,911 W 47178151532288 cmd@0x2ae8880c5220 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 16272
2017-03-27 10:05:21,911 W 47178151532288 cmd@0x2ae888000df0 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 12209
2017-03-27 10:05:21,911 W 47178151532288 cmd@0x2ae888003720 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 8022
2017-03-27 10:05:21,911 W 47178151532288 cmd@0x2ae8880013a0 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 3959
2017-03-27 10:05:21,911 W 47178151532288 cmd@0x2ae8880014a0 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 16148
2017-03-27 10:05:21,911 W 47178151532288 cmd@0x2ae888001090 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 12085
2017-03-27 10:05:21,911 W 47178151532288 cmd@0x2ae888001430 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 3373
2017-03-27 10:05:21,911 W 47178151532288 cmd@0x2ae888001280 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 7436
2017-03-27 10:05:21,911 W 47178151532288 cmd@0x2ae8880017f0 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 11491
2017-03-27 10:05:21,911 W 47178151532288 cmd@0x2ae888001880 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 15554
2017-03-27 10:05:21,911 W 47178151532288 cmd@0x2ae8880018c0 created
2017-03-27 10:05:21,911 D 47178151532288 -Keyslot = 3233
2017-03-27 10:05:21,912 W 47178151532288 cmd@0x2ae888001560 created
2017-03-27 10:05:21,912 D 47178151532288 -Keyslot = 7296
2017-03-27 10:05:21,912 W 47178151532288 cmd@0x2ae88803a630 created
2017-03-27 10:05:21,912 D 47178151532288 -Keyslot = 11367
2017-03-27 10:05:21,912 W 47178151532288 cmd@0x2ae88803a7c0 created
2017-03-27 10:05:21,912 D 47178151532288 -Keyslot = 15430
2017-03-27 10:05:21,912 W 47178151532288 cmd@0x2ae88803a730 created
2017-03-27 10:05:21,912 D 47178151532288 -Keyslot = 3109
2017-03-27 10:05:21,912 W 47178151532288 cmd@0x2ae88803a8f0 created
2017-03-27 10:05:21,914 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,915 D 47178151532288 *poll Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,915 W 47178151532288 Server(13@0x2ae8880c4b20) push _commands to _sent_commands done
2017-03-27 10:05:21,915 D 47178151532288 *poll done
2017-03-27 10:05:21,915 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,915 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,915 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,915 D 47178151532288 -Keyslot = 7172
2017-03-27 10:05:21,915 W 47178151532288 cmd@0x2ae88803a850 created
2017-03-27 10:05:21,915 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,915 D 47178151532288 *poll done
2017-03-27 10:05:21,916 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,916 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,916 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,916 D 47178151532288 -Keyslot = 5306
2017-03-27 10:05:21,916 W 47178151532288 cmd@0x2ae88803aac0 created
2017-03-27 10:05:21,917 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,917 D 47178151532288 *poll done
2017-03-27 10:05:21,920 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,920 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,920 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,920 D 47178151532288 -Keyslot = 1179
2017-03-27 10:05:21,920 W 47178151532288 cmd@0x2ae88803acb0 created
2017-03-27 10:05:21,920 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,920 D 47178151532288 *poll done
2017-03-27 10:05:21,927 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,927 D 47178151532288 *poll Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,927 D 47178151532288 Read Server(20@0x2ae8880c3d50) buffer size 2082
2017-03-27 10:05:21,928 D 47178151532288 +responses size: 2
2017-03-27 10:05:21,928 D 47178151532288 +rest buffer: 0
2017-03-27 10:05:21,928 W 47178151532288 before rsp_to, cmd@0x2ae8880c3610 in Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,928 W 47178151532288 before rsp_to, cmd@0x2ae8880c3170 in Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,929 W 47178151532288 Server(20@0x2ae8880c3d50) push _commands to _sent_commands done
2017-03-27 10:05:21,929 D 47178151532288 *poll done
2017-03-27 10:05:21,929 D 47178151532288 *poll nfds: 2
2017-03-27 10:05:21,929 D 47178151532288 *poll Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,929 D 47178151532288 Read Server(28@0x2ae8880c40b0) buffer size 2082
2017-03-27 10:05:21,929 D 47178151532288 +responses size: 2
2017-03-27 10:05:21,929 D 47178151532288 +rest buffer: 0
2017-03-27 10:05:21,929 W 47178151532288 before rsp_to, cmd@0x2ae8880c51e0 in Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,929 W 47178151532288 before rsp_to, cmd@0x2ae8880032b0 in Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,929 W 47178151532288 Server(28@0x2ae8880c40b0) push _commands to _sent_commands done
2017-03-27 10:05:21,929 D 47178151532288 *poll Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,929 D 47178151532288 Read Server(27@0x2ae8880c4460) buffer size 2082
2017-03-27 10:05:21,930 D 47178151532288 +responses size: 2
2017-03-27 10:05:21,930 D 47178151532288 +rest buffer: 0
2017-03-27 10:05:21,930 W 47178151532288 before rsp_to, cmd@0x2ae888000f70 in Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,930 W 47178151532288 before rsp_to, cmd@0x2ae8880c5140 in Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,930 W 47178151532288 Server(27@0x2ae8880c4460) push _commands to _sent_commands done
2017-03-27 10:05:21,930 D 47178151532288 *poll done
2017-03-27 10:05:21,932 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,932 D 47178151532288 *poll Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,933 D 47178151532288 Read Server(13@0x2ae8880c4b20) buffer size 2082
2017-03-27 10:05:21,933 D 47178151532288 +responses size: 2
2017-03-27 10:05:21,933 D 47178151532288 +rest buffer: 0
2017-03-27 10:05:21,933 W 47178151532288 before rsp_to, cmd@0x2ae888002f40 in Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,933 W 47178151532288 before rsp_to, cmd@0x2ae888003cb0 in Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,933 W 47178151532288 Server(13@0x2ae8880c4b20) push _commands to _sent_commands done
2017-03-27 10:05:21,933 D 47178151532288 *poll done
2017-03-27 10:05:21,933 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,933 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,934 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 26 read returns 26
2017-03-27 10:05:21,934 D 47178151532288 -Keyslot = 13684
2017-03-27 10:05:21,934 W 47178151532288 cmd@0x2ae88803a980 created
2017-03-27 10:05:21,934 D 47178151532288 Client(12@0x2ae888000aa0) Process 29 over 29 commands
2017-03-27 10:05:21,934 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae8880c52d0
2017-03-27 10:05:21,934 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,934 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae888003140
2017-03-27 10:05:21,934 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,934 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae888003410
2017-03-27 10:05:21,934 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,934 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae888000fb0
2017-03-27 10:05:21,935 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,935 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae8880c51a0
2017-03-27 10:05:21,935 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,944 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae8880c31d0
2017-03-27 10:05:21,944 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,944 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae888000eb0
2017-03-27 10:05:21,945 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,945 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae888003980
2017-03-27 10:05:21,945 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,945 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae888000e50
2017-03-27 10:05:21,945 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,945 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae8880c5220
2017-03-27 10:05:21,945 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,945 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae888000df0
2017-03-27 10:05:21,945 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,945 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae888003720
2017-03-27 10:05:21,945 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,945 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae8880013a0
2017-03-27 10:05:21,945 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,945 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae8880014a0
2017-03-27 10:05:21,945 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,945 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae888001090
2017-03-27 10:05:21,945 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,945 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae888001430
2017-03-27 10:05:21,945 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,945 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae888001280
2017-03-27 10:05:21,945 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,945 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae8880017f0
2017-03-27 10:05:21,945 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,945 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae888001880
2017-03-27 10:05:21,946 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,946 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae8880018c0
2017-03-27 10:05:21,946 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,946 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae888001560
2017-03-27 10:05:21,946 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,946 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae88803a630
2017-03-27 10:05:21,946 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,946 W 47178151532288 after Server(27@0x2ae8880c4460) push_back cmd@0x2ae88803a7c0
2017-03-27 10:05:21,946 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,946 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae88803a730
2017-03-27 10:05:21,946 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,946 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae88803a8f0
2017-03-27 10:05:21,946 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,946 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae88803a850
2017-03-27 10:05:21,946 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,946 W 47178151532288 after Server(13@0x2ae8880c4b20) push_back cmd@0x2ae88803aac0
2017-03-27 10:05:21,946 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,946 W 47178151532288 after Server(20@0x2ae8880c3d50) push_back cmd@0x2ae88803acb0
2017-03-27 10:05:21,946 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,946 W 47178151532288 after Server(28@0x2ae8880c40b0) push_back cmd@0x2ae88803a980
2017-03-27 10:05:21,946 W 47178151532288 Client(12@0x2ae888000aa0) add peer Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,946 W 47178151532288 in _process() peer,size=4
2017-03-27 10:05:21,947 D 47178151532288 Processed, rest buffer 0
2017-03-27 10:05:21,947 D 47178151532288 *writev to 12 iovcnt=8 total bytes=8328
2017-03-27 10:05:21,947 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c51e0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,947 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888000f70 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,956 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888002f40 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,956 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c3610 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,958 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c5140 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,958 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880032b0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,958 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c3170 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,958 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888003cb0 for Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,958 W 47178151532288 before clear peer,size=4
2017-03-27 10:05:21,958 D 47178151532288 *poll done
2017-03-27 10:05:21,958 D 47178151532288 *poll nfds: 5
2017-03-27 10:05:21,958 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,958 D 47178151532288 Read from Client(12@0x2ae888000aa0) current buffer size: 364 read returns 364
2017-03-27 10:05:21,958 D 47178151532288 -Keyslot = 9557
2017-03-27 10:05:21,958 W 47178151532288 cmd@0x2ae888003cb0 created
2017-03-27 10:05:21,959 D 47178151532288 -Keyslot = 5430
2017-03-27 10:05:21,959 W 47178151532288 cmd@0x2ae8880038d0 created
2017-03-27 10:05:21,959 D 47178151532288 -Keyslot = 1303
2017-03-27 10:05:21,959 W 47178151532288 cmd@0x2ae888000920 created
2017-03-27 10:05:21,959 D 47178151532288 -Keyslot = 13808
2017-03-27 10:05:21,959 W 47178151532288 cmd@0x2ae888002b60 created
2017-03-27 10:05:21,959 D 47178151532288 -Keyslot = 9681
2017-03-27 10:05:21,959 W 47178151532288 cmd@0x2ae888002ba0 created
2017-03-27 10:05:21,959 D 47178151532288 -Keyslot = 5554
2017-03-27 10:05:21,959 W 47178151532288 cmd@0x2ae8880c3170 created
2017-03-27 10:05:21,959 D 47178151532288 -Keyslot = 1427
2017-03-27 10:05:21,959 W 47178151532288 cmd@0x2ae8880032b0 created
2017-03-27 10:05:21,959 D 47178151532288 -Keyslot = 10123
2017-03-27 10:05:21,959 W 47178151532288 cmd@0x2ae8880c5140 created
2017-03-27 10:05:21,959 D 47178151532288 -Keyslot = 14250
2017-03-27 10:05:21,959 W 47178151532288 cmd@0x2ae8880c3610 created
2017-03-27 10:05:21,959 D 47178151532288 -Keyslot = 1605
2017-03-27 10:05:21,959 W 47178151532288 cmd@0x2ae888002f40 created
2017-03-27 10:05:21,959 D 47178151532288 -Keyslot = 5732
2017-03-27 10:05:21,959 W 47178151532288 cmd@0x2ae888000f70 created
2017-03-27 10:05:21,959 D 47178151532288 -Keyslot = 9735
2017-03-27 10:05:21,959 W 47178151532288 cmd@0x2ae8880c51e0 created
2017-03-27 10:05:21,959 D 47178151532288 -Keyslot = 13862
2017-03-27 10:05:21,960 W 47178151532288 cmd@0x2ae888002be0 created
2017-03-27 10:05:21,960 D 47178151532288 -Keyslot = 1729
2017-03-27 10:05:21,960 W 47178151532288 cmd@0x2ae888002c20 created
2017-03-27 10:05:21,960 E 47178151532288 Client(12@0x2ae888000aa0) _awaiting_groups still not-empty
2017-03-27 10:05:21,960 D 47178151532288 *poll Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,965 W 47178151532288 Server(20@0x2ae8880c3d50) push _commands to _sent_commands done
2017-03-27 10:05:21,965 D 47178151532288 *writev to 20 iovcnt=7 total bytes=182
2017-03-27 10:05:21,965 D 47178151532288 *poll Server(28@0x2ae8880c40b0)
2017-03-27 10:05:21,965 W 47178151532288 Server(28@0x2ae8880c40b0) push _commands to _sent_commands done
2017-03-27 10:05:21,965 D 47178151532288 *writev to 28 iovcnt=8 total bytes=208
2017-03-27 10:05:21,965 D 47178151532288 *poll Server(27@0x2ae8880c4460)
2017-03-27 10:05:21,965 W 47178151532288 Server(27@0x2ae8880c4460) push _commands to _sent_commands done
2017-03-27 10:05:21,965 D 47178151532288 *writev to 27 iovcnt=7 total bytes=182
2017-03-27 10:05:21,965 D 47178151532288 *poll Server(13@0x2ae8880c4b20)
2017-03-27 10:05:21,966 W 47178151532288 Server(13@0x2ae8880c4b20) push _commands to _sent_commands done
2017-03-27 10:05:21,966 D 47178151532288 *writev to 13 iovcnt=7 total bytes=182
2017-03-27 10:05:21,966 D 47178151532288 *poll done
2017-03-27 10:05:21,973 D 47178151532288 *poll nfds: 1
2017-03-27 10:05:21,973 D 47178151532288 *poll Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,973 D 47178151532288 client::on_events IOError: buffer read Connection reset by peer :: Close Client(12@0x2ae888000aa0)
2017-03-27 10:05:21,974 D 47178151532288 CLOSE fd=12
2017-03-27 10:05:21,974 W 47178151532288 Client(-1@0x2ae888000aa0) closed. delete it
2017-03-27 10:05:21,974 W 47178151532288 Client(-1@0x2ae888000aa0) start destructing,peer size=0
2017-03-27 10:05:21,974 W 47178151532288 Client(-1@0x2ae888000aa0) end destructing
2017-03-27 10:05:21,974 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c52d0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,974 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888003140 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,974 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888003410 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,974 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888000fb0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,974 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c51a0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c31d0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888000eb0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888003980 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888000e50 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c5220 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888000df0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888003720 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880013a0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880014a0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888001090 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888001430 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888001280 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880017f0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888001880 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880018c0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888001560 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,977 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88803a630 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88803a7c0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88803a730 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88803a8f0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88803a850 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88803aac0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88803acb0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae88803a980 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888003cb0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880038d0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888000920 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888002b60 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888002ba0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c3170 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880032b0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c5140 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c3610 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888002f40 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,978 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888000f70 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,979 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae8880c51e0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,979 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888002be0 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,979 W 47178151532288 ~SingleCommandGroup() called,cmd@0x2ae888002c20 for Client(-1@0x2ae888000aa0)
2017-03-27 10:05:21,991 D 47178151532288 *poll done
2017-03-27 10:05:21,991 D 47178151532288 *poll nfds: 4
2017-03-27 10:05:21,991 D 47178151532288 *poll Server(20@0x2ae8880c3d50)
2017-03-27 10:05:21,991 D 47178151532288 Read Server(20@0x2ae8880c3d50) buffer size 7287
2017-03-27 10:05:21,992 D 47178151532288 +responses size: 7
2017-03-27 10:05:21,992 D 47178151532288 +rest buffer: 0
2017-03-27 10:05:21,992 W 47178151532288 before rsp_to, cmd@0x2ae888003410 in Server(20@0x2ae8880c3d50)
仔细分析日志可知在cerberus在旧命令的响应到来后把_peers变量清空了，而新命令的_peers在此之前就已经添加到了_peers里，结果客户端一旦突然断开，_peers为空，无法在Client析构方法里处理_sent_commands里失效的命令；等再处理_sent_commands命令时，command已经被析构。