How to balance the cassandra cluster - cassandra
We have 30 hadoop prod nodes and they the cluster is unbalanced.
Datacenter: Hadoop
==================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns Host ID Rack
UN x.x.x.1 13.64 GB 1 0.0% 16e90f33-6f79-41e8-bd25-22e5eea6707b RAC-1
UN x.x.x.2 11.27 GB 1 0.0% 3a9da644-2587-42b3-9162-7fb504cfbf97 RAC-3
UN x.x.x.3 10.76 GB 1 0.0% e0b67015-bb37-466b-b837-17d758535a4d RAC-1
UN x.x.x.4 10.07 GB 1 0.0% 2110d060-3d6f-404a-8979-a4c98b1e63ae RAC-1
UN x.x.x.5 260.31 GB 1 0.0% 48da7d4d-f396-4d12-a481-c316b367194b RAC-1
UN x.x.x.6 17.32 GB 1 0.0% 45806806-5c84-4898-9835-9e0874a48ded RAC-2
UN x.x.x.7 7.66 GB 1 0.0% 2724dddf-011e-431f-8936-0dc576ee833e RAC-2
UN x.x.x.8 11.95 GB 1 0.0% 61e83114-32ad-4abe-b6bf-228792bcd0fa RAC-2
UN x.x.x.9 16.38 GB 1 0.0% 988ec834-c4e5-4170-9c04-c5a7d20b3094 RAC-3
UN x.x.x.10 10.53 GB 1 0.0% 0b53cf24-541f-4e25-810b-7020e10817ee RAC-3
UN x.x.x.11 10.3 GB 1 0.0% ae14518d-c1c2-4c21-998b-273e961cd1c0 RAC-2
UN x.x.x.12 15.42 GB 1 0.0% 019f1a17-11c6-4a38-b7f9-658030fe0eac RAC-2
UN x.x.x.13 11.1 GB 1 0.0% 0cdabec3-95e2-4451-ba69-a659874fe300 RAC-2
UN x.x.x.14 9.58 GB 1 0.0% 4c43064c-be29-4723-be2d-cd0e3cd92909 RAC-3
UN x.x.x.15 11.73 GB 1 0.0% cc469cee-1ca0-45a9-bfd0-c182b0727238 RAC-2
UN x.x.x.16 9.55 GB 1 0.0% 0ccd790e-7d1c-4cc8-8ebb-9ec786ee9962 RAC-2
UN x.x.x.17 9.44 GB 1 0.0% 3a60244e-8af9-45a4-988a-d5158fbe04e7 RAC-3
UN x.x.x.18 11.44 GB 1 0.0% 0b7508e9-e06f-4532-841c-08047f6ebf35 RAC-3
UN x.x.x.19 13.25 GB 1 0.0% 7648792b-9c92-45be-b82a-e171d21756d6 RAC-3
UN x.x.x.20 256.81 GB 1 0.0% 92033ad7-d60f-4a89-9439-0af7e744a246 RAC-2
UN x.x.x.21 10.03 GB 1 0.0% e494a90f-64b1-4f84-94fa-b228a8ef3160 RAC-1
UN x.x.x.22 9.32 GB 1 0.0% 64f9a2e4-2aab-408c-9d5f-5867ab26398c RAC-3
UN x.x.x.23 14.74 GB 1 0.0% 0ea50e73-b36c-44e9-934f-a92f14acbe23 RAC-1
UN x.x.x.24 12.37 GB 1 0.0% 804927a6-d096-4b6e-92af-43ad13e10504 RAC-1
UN x.x.x.24 258.85 GB 1 0.0% 7c1bc96c-4806-4216-bd6a-a28db4b528d1 RAC-3
UN x.x.x.26 10.38 GB 1 0.0% 2932bae4-c656-4378-9570-0f79131fe3a8 RAC-2
UN x.x.x.27 11.67 GB 1 0.0% 918bc253-40b6-4a56-ab4a-e50953ffb355 RAC-1
UN x.x.x.28 8.17 GB 1 0.0% bb302317-3671-4174-82b0-43f53c683f44 RAC-1
UN x.x.x.29 10.57 GB 1 0.0% ff4a1e2e-249b-44d7-b488-0acd99a6db86 RAC-1
UN x.x.x.30 12.27 GB 1 0.0% df75362f-24c0-4783-a03c-b2a578c0927d RAC-3
Almost all the data is going to only 3 nodes(5,20 and 24) in the cluster. I cannot use vnodes because DSE hadoop nodes not support it.
I know we can use nodetool move to rearrange the tokens but how to calculate the middle token?
How to balance it? how to calculate the tokens and move it?
Related
Why is docker stats CPU Percentage greater than 100 times number of cores
I have an Azure VM with 2 cores. From my understanding, the CPU % returned by docker stats can be greater than 100% if multiple cores are used. So, this should max out at 200% for this VM. However, I get results like this with CPU % greater than 1000% CONTAINER CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS 545d4c69028f 3.54% 94.39 MiB / 6.803 GiB 1.35% 3.36 MB / 1.442 MB 1.565 MB / 5.673 MB 6 008893e3f70c 625.00% 191.3 MiB / 6.803 GiB 2.75% 0 B / 0 B 0 B / 24.58 kB 35 f49c94dc4567 0.10% 46.85 MiB / 6.803 GiB 0.67% 2.614 MB / 5.01 MB 61.44 kB / 0 B 31 08415d81c355 0.00% 28.76 MiB / 6.803 GiB 0.41% 619.1 kB / 3.701 MB 0 B / 0 B 11 03f54d35a5f8 1.04% 136.5 MiB / 6.803 GiB 1.96% 83.94 MB / 7.721 MB 0 B / 0 B 22 f92faa7321d8 0.15% 19.29 MiB / 6.803 GiB 0.28% 552.5 kB / 758.6 kB 0 B / 2.798 MB 7 2f4a27cc3e44 0.07% 303.8 MiB / 6.803 GiB 4.36% 32.52 MB / 20.27 MB 2.195 MB / 0 B 11 ac96bc45044a 0.00% 19.34 MiB / 6.803 GiB 0.28% 37.28 kB / 12.76 kB 0 B / 3.633 MB 7 7c1a45e92f52 2.20% 356.9 MiB / 6.803 GiB 5.12% 86.36 MB / 156.2 MB 806.9 kB / 0 B 16 0bc4f319b721 14.98% 101.8 MiB / 6.803 GiB 1.46% 138.1 MB / 64.33 MB 0 B / 73.74 MB 75 66aa24598d27 2269.46% 1.269 GiB / 6.803 GiB 18.65% 1.102 GB / 256.4 MB 14.34 MB / 3.412 MB 50 I can verify there are only two cores: $ grep -c ^processor /proc/cpuinfo 2 The output of lshw -short is also confusing to me: H/W path Device Class Description ===================================================== system Virtual Machine /0 bus Virtual Machine /0/0 memory 64KiB BIOS /0/5 processor Intel(R) Xeon(R) CPU E5-2673 v3 # 2.40GHz /0/6 processor Xeon (None) /0/7 processor (None) /0/8 processor (None) /0/9 processor (None) /0/a processor (None) /0/b processor (None) /0/c processor (None) /0/d processor (None) /0/e processor (None) /0/f processor (None) /0/10 processor (None) ... with well over 50 processors listed
For your first question, I would suggest you to submit an issue on this page. The output of lshw -short is also confusing to me: If you omit the "-short" parameter, you will find that all of the "processor (None)" is in the state of DISABLED.
boost::asio::io_service performance loss when network card set irq affinity
I've been testing the performance(qps) of pbrpc framework(sofa-pbrpc). When I used cpu0 dealing software interrupt, the qps is 100w+. top command top - 18:00:54 up 158 days, 18 min, 3 users, load average: 18.09, 19.29, 18.74 Tasks: 793 total, 3 running, 790 sleeping, 0 stopped, 0 zombie Cpu0 : 0.0% us, 0.0% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 100.0% si Cpu1 : 58.7% us, 11.6% sy, 0.7% ni, 28.4% id, 0.7% wa, 0.0% hi, 0.0% si Cpu2 : 68.0% us, 11.6% sy, 0.0% ni, 19.8% id, 0.7% wa, 0.0% hi, 0.0% si Cpu3 : 52.3% us, 8.3% sy, 0.0% ni, 39.1% id, 0.3% wa, 0.0% hi, 0.0% si Cpu4 : 73.5% us, 11.9% sy, 0.0% ni, 13.9% id, 0.7% wa, 0.0% hi, 0.0% si Cpu5 : 52.5% us, 10.2% sy, 0.3% ni, 36.6% id, 0.3% wa, 0.0% hi, 0.0% si Cpu6 : 72.4% us, 13.2% sy, 0.0% ni, 13.2% id, 1.3% wa, 0.0% hi, 0.0% si Cpu7 : 74.7% us, 11.8% sy, 0.0% ni, 12.5% id, 1.0% wa, 0.0% hi, 0.0% si Cpu8 : 72.5% us, 12.8% sy, 0.0% ni, 14.1% id, 0.7% wa, 0.0% hi, 0.0% si Cpu9 : 82.9% us, 13.5% sy, 0.0% ni, 3.0% id, 0.7% wa, 0.0% hi, 0.0% si Cpu10 : 82.1% us, 14.6% sy, 0.0% ni, 3.0% id, 0.3% wa, 0.0% hi, 0.0% si Cpu11 : 85.9% us, 13.2% sy, 0.0% ni, 0.7% id, 0.3% wa, 0.0% hi, 0.0% si Cpu12 : 84.2% us, 14.5% sy, 0.0% ni, 0.0% id, 1.3% wa, 0.0% hi, 0.0% si Cpu13 : 67.4% us, 12.2% sy, 0.0% ni, 19.7% id, 0.7% wa, 0.0% hi, 0.0% si Cpu14 : 72.6% us, 13.5% sy, 0.0% ni, 13.2% id, 0.7% wa, 0.0% hi, 0.0% si Cpu15 : 77.4% us, 12.8% sy, 0.0% ni, 8.9% id, 1.0% wa, 0.0% hi, 0.0% si Cpu16 : 84.5% us, 14.5% sy, 0.0% ni, 0.3% id, 0.7% wa, 0.0% hi, 0.0% si Cpu17 : 84.1% us, 14.2% sy, 0.0% ni, 1.0% id, 0.7% wa, 0.0% hi, 0.0% si Cpu18 : 84.1% us, 13.6% sy, 0.0% ni, 2.0% id, 0.3% wa, 0.0% hi, 0.0% si Cpu19 : 83.2% us, 14.2% sy, 0.0% ni, 1.7% id, 1.0% wa, 0.0% hi, 0.0% si Cpu20 : 28.7% us, 4.3% sy, 0.0% ni, 63.7% id, 3.3% wa, 0.0% hi, 0.0% si Cpu21 : 83.2% us, 13.9% sy, 0.0% ni, 2.3% id, 0.7% wa, 0.0% hi, 0.0% si Cpu22 : 1.0% us, 2.3% sy, 0.3% ni, 96.4% id, 0.0% wa, 0.0% hi, 0.0% si Cpu23 : 0.3% us, 0.3% sy, 0.3% ni, 99.0% id, 0.0% wa, 0.0% hi, 0.0% si Cpu24 : 84.8% us, 13.2% sy, 0.0% ni, 1.0% id, 1.0% wa, 0.0% hi, 0.0% si Cpu25 : 84.8% us, 14.6% sy, 0.0% ni, 0.0% id, 0.7% wa, 0.0% hi, 0.0% si Cpu26 : 84.8% us, 14.5% sy, 0.0% ni, 0.0% id, 0.7% wa, 0.0% hi, 0.0% si Cpu27 : 83.4% us, 14.9% sy, 0.0% ni, 0.7% id, 1.0% wa, 0.0% hi, 0.0% si Cpu28 : 83.9% us, 14.8% sy, 0.0% ni, 0.3% id, 1.0% wa, 0.0% hi, 0.0% si Cpu29 : 23.1% us, 3.6% sy, 0.0% ni, 72.9% id, 0.3% wa, 0.0% hi, 0.0% si Cpu30 : 2.0% us, 0.7% sy, 0.3% ni, 97.0% id, 0.0% wa, 0.0% hi, 0.0% si Cpu31 : 3.9% us, 1.3% sy, 0.0% ni, 94.7% id, 0.0% wa, 0.0% hi, 0.0% si Mem: 132133912k total, 4835356k used, 127298556k free, 153412k buffers However, using set_irq_affinity.sh, the qps degraded to 70w+. top command top - 18:05:38 up 158 days, 22 min, 3 users, load average: 13.13, 18.03, 18.51 Tasks: 801 total, 2 running, 798 sleeping, 1 stopped, 0 zombie Cpu0 : 62.2% us, 13.2% sy, 0.0% ni, 4.6% id, 0.7% wa, 0.0% hi, 19.4% si Cpu1 : 55.8% us, 10.9% sy, 0.3% ni, 15.5% id, 0.0% wa, 0.0% hi, 17.5% si Cpu2 : 59.5% us, 12.5% sy, 0.0% ni, 12.8% id, 0.3% wa, 0.0% hi, 14.8% si Cpu3 : 33.7% us, 9.6% sy, 0.0% ni, 47.9% id, 0.3% wa, 0.0% hi, 8.6% si Cpu4 : 59.1% us, 12.2% sy, 0.0% ni, 11.2% id, 0.7% wa, 0.0% hi, 16.8% si Cpu5 : 51.0% us, 10.5% sy, 0.0% ni, 27.0% id, 0.3% wa, 0.0% hi, 11.2% si Cpu6 : 66.9% us, 12.8% sy, 0.0% ni, 3.6% id, 0.7% wa, 0.0% hi, 16.1% si Cpu7 : 64.8% us, 12.8% sy, 0.0% ni, 3.0% id, 0.3% wa, 0.0% hi, 19.1% si Cpu8 : 48.0% us, 9.2% sy, 0.3% ni, 32.2% id, 0.3% wa, 0.0% hi, 9.9% si Cpu9 : 50.0% us, 10.3% sy, 0.0% ni, 25.2% id, 0.3% wa, 0.0% hi, 14.2% si Cpu10 : 63.0% us, 11.9% sy, 0.0% ni, 7.9% id, 0.3% wa, 0.0% hi, 16.8% si Cpu11 : 60.9% us, 12.8% sy, 0.0% ni, 8.9% id, 0.3% wa, 0.0% hi, 17.1% si Cpu12 : 62.6% us, 12.6% sy, 0.0% ni, 3.6% id, 0.3% wa, 0.0% hi, 20.9% si Cpu13 : 56.2% us, 11.8% sy, 0.0% ni, 19.1% id, 0.3% wa, 0.0% hi, 12.5% si Cpu14 : 64.0% us, 13.2% sy, 0.0% ni, 6.9% id, 0.3% wa, 0.0% hi, 15.5% si Cpu15 : 65.0% us, 14.5% sy, 0.0% ni, 3.3% id, 0.7% wa, 0.0% hi, 16.5% si Cpu16 : 65.8% us, 12.5% sy, 0.0% ni, 1.6% id, 0.3% wa, 0.0% hi, 19.7% si Cpu17 : 63.2% us, 12.9% sy, 0.0% ni, 2.6% id, 0.3% wa, 0.0% hi, 20.9% si Cpu18 : 15.2% us, 4.0% sy, 0.0% ni, 77.6% id, 0.0% wa, 0.0% hi, 3.3% si Cpu19 : 58.4% us, 12.2% sy, 0.0% ni, 12.9% id, 0.3% wa, 0.0% hi, 16.2% si Cpu20 : 48.8% us, 10.2% sy, 0.0% ni, 27.4% id, 0.3% wa, 0.0% hi, 13.2% si Cpu21 : 31.6% us, 5.9% sy, 0.0% ni, 54.3% id, 0.3% wa, 0.0% hi, 7.9% si Cpu22 : 0.7% us, 1.3% sy, 0.0% ni, 98.0% id, 0.0% wa, 0.0% hi, 0.0% si Cpu23 : 64.8% us, 12.8% sy, 0.0% ni, 1.6% id, 0.7% wa, 0.0% hi, 20.1% si Cpu24 : 64.8% us, 12.2% sy, 0.0% ni, 5.3% id, 0.7% wa, 0.0% hi, 17.1% si Cpu25 : 59.5% us, 14.1% sy, 0.0% ni, 8.9% id, 0.3% wa, 0.0% hi, 17.1% si Cpu26 : 63.7% us, 13.5% sy, 0.0% ni, 3.0% id, 0.0% wa, 0.0% hi, 19.8% si Cpu27 : 60.7% us, 12.2% sy, 0.0% ni, 9.2% id, 0.3% wa, 0.0% hi, 17.5% si Cpu28 : 62.2% us, 13.5% sy, 0.0% ni, 1.3% id, 0.3% wa, 0.0% hi, 22.7% si Cpu29 : 12.5% us, 4.6% sy, 0.3% ni, 79.2% id, 0.0% wa, 0.0% hi, 3.3% si Cpu30 : 2.3% us, 2.0% sy, 1.3% ni, 94.1% id, 0.0% wa, 0.0% hi, 0.3% si Cpu31 : 3.3% us, 1.7% sy, 0.0% ni, 94.7% id, 0.0% wa, 0.0% hi, 0.3% si Mem: 132133912k total, 4888040k used, 127245872k free, 153440k buffers Is there some special handling in boost asio？
CPU Higher than expected in Node running in docker
I have a vagrant machine running at 33% CPU on my Mac (10.9.5) when nothing is supposed to be happening. The VM machine is run by Kinematic. Looking inside one of the containers I see 2 node (v0.12.2) processes running at 3-4% CPU each.
root#49ab3ab54901:/usr/src# top -bc
top - 03:11:59 up 8:31, 0 users, load average: 0.13, 0.18, 0.22
Tasks: 7 total, 1 running, 6 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.2 us, 0.7 sy, 0.0 ni, 99.1 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem: 2051824 total, 1942836 used, 108988 free, 74572 buffers
KiB Swap: 1466848 total, 18924 used, 1447924 free. 326644 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1 root 20 0 4332 672 656 S 0.0 0.0 0:00.10 /bin/sh -c node -e "require('./seed/seeder.js').seed().then(function (resp) { console.log('successfully seeded!'); pro+
15 root 20 0 737320 81008 13452 S 0.0 3.9 0:32.57 node /usr/local/bin/nodemon app/api.js
33 root 20 0 4332 740 652 S 0.0 0.0 0:00.00 sh -c node app/api.js
34 root 20 0 865080 68952 14244 S 0.0 3.4 0:01.70 node app/api.js
83 root 20 0 20272 3288 2776 S 0.0 0.2 0:00.11 bash
18563 root 20 0 20248 3152 2840 S 0.0 0.2 0:00.11 bash
18575 root 20 0 21808 2308 2040 R 0.0 0.1 0:00.00 top -bc
I went on and runned a node --prof and processed the log with node-tick-processor. It looks like that 99.3% of CPU is used in the syscall :
(for full output see http://pastebin.com/6qgFuFWK )
root#d6d78487e1ec:/usr/src# node-tick-processor isolate-0x26c0180-v8.log
...
Statistical profiling result from isolate-0x26c0180-v8.log, (130664 ticks, 0 unaccounted, 0 excluded).
...
[C++]:
ticks total nonlib name
129736 99.3% 99.3% syscall
160 0.1% 0.1% node::ContextifyScript::New(v8::FunctionCallbackInfo<v8::Value> const&)
124 0.1% 0.1% __write
73 0.1% 0.1% __xstat
18 0.0% 0.0% v8::internal::Heap::AllocateFixedArray(int, v8::internal::PretenureFlag)
18 0.0% 0.0% node::Stat(v8::FunctionCallbackInfo<v8::Value> const&)
17 0.0% 0.0% __lxstat
16 0.0% 0.0% node::Read(v8::FunctionCallbackInfo<v8::Value> const&)
...
1 0.0% 0.0% __fxstat
1 0.0% 0.0% _IO_default_xsputn
[GC]:
ticks total nonlib name
22 0.0%
[Bottom up (heavy) profile]:
Note: percentage shows a share of a particular caller in the total
amount of its parent calls.
Callers occupying less than 2.0% are not shown.
ticks parent name
129736 99.3% syscall
[Top down (heavy) profile]:
Note: callees occupying less than 0.1% are not shown.
inclusive self name
ticks total ticks total
129736 99.3% 129736 99.3% syscall
865 0.7% 0 0.0% Function: ~<anonymous> node.js:27:10
864 0.7% 0 0.0% LazyCompile: ~startup node.js:30:19
851 0.7% 0 0.0% LazyCompile: ~Module.runMain module.js:499:26
799 0.6% 0 0.0% LazyCompile: Module._load module.js:273:24
795 0.6% 0 0.0% LazyCompile: ~Module.load module.js:345:33
794 0.6% 0 0.0% LazyCompile: ~Module._extensions..js module.js:476:37
792 0.6% 0 0.0% LazyCompile: ~Module._compile module.js:378:37
791 0.6% 0 0.0% Function: ~<anonymous> /usr/src/app/api.js:1:11
791 0.6% 0 0.0% LazyCompile: ~require module.js:383:19
791 0.6% 0 0.0% LazyCompile: ~Module.require module.js:362:36
791 0.6% 0 0.0% LazyCompile: Module._load module.js:273:24
788 0.6% 0 0.0% LazyCompile: ~Module.load module.js:345:33
786 0.6% 0 0.0% LazyCompile: ~Module._extensions..js module.js:476:37
783 0.6% 0 0.0% LazyCompile: ~Module._compile module.js:378:37
644 0.5% 0 0.0% Function: ~<anonymous> /usr/src/app/api.authentication.js:1:11
627 0.5% 0 0.0%
...
A strace resulted in nothing abnormal:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
54.51 0.001681 76 22 clone
17.28 0.000533 4 132 epoll_ctl
16.80 0.000518 24 22 wait4
6.39 0.000197 2 110 66 stat
5.03 0.000155 1 176 close
0.00 0.000000 0 176 read
0.00 0.000000 0 88 write
0.00 0.000000 0 44 rt_sigaction
0.00 0.000000 0 88 rt_sigprocmask
0.00 0.000000 0 22 rt_sigreturn
0.00 0.000000 0 66 ioctl
0.00 0.000000 0 66 socketpair
0.00 0.000000 0 88 epoll_wait
0.00 0.000000 0 22 pipe2
------ ----------- ----------- --------- --------- ----------------
100.00 0.003084 1122 66 total
And the other node process:
% time seconds usecs/call calls errors syscall
------ ----------- ----------- --------- --------- ----------------
0.00 0.000000 0 14 epoll_wait
------ ----------- ----------- --------- --------- ----------------
100.00 0.000000 14 total
Am I missing something?
I wonder if it is VirtualBox's or Docker's layers consuming 4%.
When you have a few containers with 2 processes running at 4%, it adds up quickly.
Cryptic Node Tick Result -- CPU Taking 100%
So ive run into a slight snag with an application that I am writing. The application is a simple relay server, using Socket.IO and Node. Everything works great, but when it gets under heavy load, the process spikes to 100% CPU and stays there. I ran a node-tick on it, and this is what the result was Statistical profiling result from v8.log, (83336 ticks, 0 unaccounted, 0 excluded). [Shared libraries]: ticks total nonlib name 83220 99.9% 0.0% /lib/x86_64-linux-gnu/libc-2.19.so 97 0.1% 0.0% /usr/bin/nodejs 9 0.0% 0.0% /lib/x86_64-linux-gnu/libpthread-2.19.so [JavaScript]: ticks total nonlib name 1 0.0% 10.0% Stub: FastNewClosureStub 1 0.0% 10.0% Stub: CallConstructStub_Recording 1 0.0% 10.0% LazyCompile: ~stringify native json.js:308 1 0.0% 10.0% LazyCompile: ~hash /opt/connect/node/node_modules/sticky-session/lib/sticky-session.js:4 1 0.0% 10.0% LazyCompile: ~exports.dirname path.js:415 1 0.0% 10.0% LazyCompile: ~buildFn /opt/connect/node/node_modules/socket.io-redis/node_modules/msgpack-js/node_modules/bops/read.js:5 1 0.0% 10.0% LazyCompile: ~addListener events.js:126 1 0.0% 10.0% LazyCompile: ~ToObject native runtime.js:567 1 0.0% 10.0% LazyCompile: *setTime native date.js:482 1 0.0% 10.0% LazyCompile: *DefineOneShotAccessor native messages.js:767 [C++]: ticks total nonlib name [GC]: ticks total nonlib name 3 0.0% [Bottom up (heavy) profile]: Note: percentage shows a share of a particular caller in the total amount of its parent calls. Callers occupying less than 2.0% are not shown. ticks parent name 83220 99.9% /lib/x86_64-linux-gnu/libc-2.19.so [Top down (heavy) profile]: Note: callees occupying less than 0.1% are not shown. inclusive self name ticks total ticks total 83192 99.8% 83192 99.8% /lib/x86_64-linux-gnu/libc-2.19.so 114 0.1% 0 0.0% Function: ~<anonymous> node.js:27 113 0.1% 0 0.0% LazyCompile: ~startup node.js:30 103 0.1% 0 0.0% LazyCompile: ~Module.runMain module.js:495 98 0.1% 0 0.0% LazyCompile: Module._load module.js:275 90 0.1% 0 0.0% LazyCompile: ~Module.load module.js:346 89 0.1% 0 0.0% LazyCompile: ~Module._extensions..js module.js:472 88 0.1% 0 0.0% LazyCompile: ~Module._compile module.js:374 88 0.1% 0 0.0% Function: ~<anonymous> /opt/connect/node/lib/connect.js:1 83220 99.9% 0.0% /lib/x86_64-linux-gnu/libc-2.19.so Only 1 library seems to be the problem -- but I can figure out what the heck it is for. My hunch is the json processing im doing -- but I can be sure. Has anyone run into this problem before?
When I using MTR, why farther node values lower?
The mtr report like this: shell> mtr --report ec2-122-248-229-83.ap-southeast-1.compute.amazonaws.com HOST: macserver.local Loss% Snt Last Avg Best Wrst StDev 1.|-- 192.168.12.1 0.0% 10 1.2 2.9 0.9 7.4 2.3 2.|-- 101.36.89.49 0.0% 10 6.8 5.7 2.1 16.6 4.3 3.|-- 192.168.17.37 0.0% 10 53.8 164.9 4.9 904.4 304.0 4.|-- 220.181.105.25 0.0% 10 5.1 11.1 5.1 26.9 7.1 5.|-- 220.181.0.5 0.0% 10 68.5 15.1 4.9 68.5 19.4 6.|-- 220.181.0.41 0.0% 10 12.6 10.2 5.0 27.1 6.5 7.|-- 202.97.53.82 0.0% 10 7.2 9.9 4.9 28.1 6.7 8.|-- 202.97.58.94 0.0% 10 16.5 10.0 5.2 16.5 3.9 9.|-- 202.97.61.98 0.0% 10 49.2 46.4 39.0 76.7 11.2 10.|-- 202.97.121.98 0.0% 10 41.1 43.5 41.1 46.3 1.6 11.|-- 63-218-213-206.static.pcc 0.0% 10 87.2 77.6 70.3 92.2 7.4 12.|-- 203.83.223.62 0.0% 10 71.9 74.8 69.9 87.2 5.1 13.|-- 203.83.223.77 0.0% 10 73.6 73.8 70.2 80.9 3.0 14.|-- ec2-175-41-128-238.ap-sou 0.0% 10 70.4 73.9 70.4 84.1 4.0 15.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0 16.|-- ec2-122-248-229-83.ap-sou 10.0% 10 82.3 76.0 70.6 88.7 6.1 Why is the average of the 16 lines lower than line 11?
Routers are designed to route packets as quickly as possible. They're not designed to generate and transmit ICMP errors as quickly as possible. Apparently, the machine at line 11 is very slow at generating ICMP errors. When you see a lower time past a hop than at that hop, you know that most likely it took that router a significant amount of time to generate an ICMP error and get it going back to you. And, of course, you have to ignore line 15. You didn't get any replies from that router.