I'm investigating a webapp which ran up to 10gb of memory, by analysing a memory dump using Windbg.
Here's the bottom of the !dumpheap -stat output:
00007ff9545df5d0 166523 13321840 System.Runtime.Caching.MemoryCache
00007ff9545df4a0 166523 14654024 System.Runtime.Caching.CacheMemoryMonitor
00007ff9545de990 166523 14654024 System.Runtime.Caching.SRef[]
00007ff9545dcef0 166523 14654024 System.Runtime.Caching.GCHandleRef`1[[System.Runtime.Caching.MemoryCacheStore, System.Runtime.Caching]][]
00007ff9545dfb28 166523 19982760 System.Runtime.Caching.MemoryCacheStatistics
00007ff956778510 333059 21315680 System.Int64[]
00007ff95679c988 41597 31250111 System.Byte[]
00007ff9545e08c8 1332184 31972416 System.Runtime.Caching.MemoryCacheEqualityComparer
00007ff9545dfe48 1332184 31972416 System.Runtime.Caching.SRef
00007ff956780ff0 1332200 31972800 System.SizedReference
00007ff956724620 1498777 35970648 System.Threading.TimerHolder
00007ff95677fb30 1536170 36868080 System.Runtime.Remoting.Messaging.CallContextSecurityData
00007ff956796f28 1606960 38567040 System.Object
00007ff9545df810 1332184 42629888 System.Runtime.Caching.GCHandleRef`1[[System.Runtime.Caching.MemoryCacheStore, System.Runtime.Caching]]
00007ff9545dda38 1332184 42629888 System.Runtime.Caching.UsageBucket[]
00007ff9567ae930 1332268 42632576 Microsoft.Win32.SafeHandles.SafeWaitHandle
00007ff9545df968 1498707 47958624 System.Runtime.Caching.GCHandleRef`1[[System.Threading.Timer, mscorlib]]
00007ff9567adbf8 1498777 47960864 System.Threading.Timer
00007ff9545dff50 1332184 53287360 System.Runtime.Caching.CacheUsage
00007ff94986ead8 1536137 61445480 System.Web.Hosting.AspNetHostExecutionContextManager+AspNetHostExecutionContext
00007ff9567a2838 1332210 63946080 System.Threading.ManualResetEvent
00007ff956796948 293525 66384986 System.String
00007ff9545dfef0 1332184 74602304 System.Runtime.Caching.CacheExpires
00007ff9567add20 1498760 95920640 System.Threading.TimerCallback
00007ff9545dfa90 1332184 106574720 System.Runtime.Caching.MemoryCacheStore
00007ff95679b3b0 1333289 106663120 System.Collections.Hashtable
00007ff95678f138 1536171 110604312 System.Runtime.Remoting.Messaging.LogicalCallContext
00007ff9545dffb0 1332184 127889664 System.Runtime.Caching.UsageBucket
00007ff95679d1e0 1333292 128664768 System.Collections.Hashtable+bucket[]
00007ff9567245c0 1498777 131892376 System.Threading.TimerQueueTimer
00007ff9567aec48 1536255 135190440 System.Threading.ExecutionContext
00007ff9545dcf78 1332184 351696576 System.Runtime.Caching.ExpiresBucket[]
000000f82c79d9f0 473339 385303992 Free
00007ff956799220 40309535 1617342672 System.Int32[]
00007ff9545e0468 39965520 3836689920 System.Runtime.Caching.ExpiresBucket
So there are nearly 40 million instances of System.Runtime.Caching.ExpiresBucket, totally nearly 4gb of the used memory. System.Runtime.Caching classes appear quite a lot in the top offenders.
I took a random instance of a System.Runtime.Caching.ExpiresBucket class, and did a !gcroot on it. It took ages (maybe 30 mins) to produce 1 thread...there may have been more but I interrupted the operation at this point.
The chain of references is over 1.5 million lines long! But I can show the important bits here:
0:000> !gcroot 000000f82dd4bc28
Thread 1964:
000000fcbdbce6a0 00007ff8f9bbe388 Microsoft.AspNet.SignalR.SqlServer.ObservableDbOperation.ExecuteReaderWithUpdates(System.Action`2<System.Data.IDataRecord,Microsoft.AspNet.SignalR.SqlServer.DbOperation>)
rbp-58: 000000fcbdbce6e8
-> 000000fa2d1f26a0 Microsoft.AspNet.SignalR.SqlServer.ObservableDbOperation+<>c__DisplayClass1e
-> 000000fa2d1f2110 Microsoft.AspNet.SignalR.SqlServer.ObservableDbOperation
-> 000000fa2d1f24d0 System.Action
-> 000000fa2d1f24a8 System.Object[]
-> 000000fa2d1f2468 System.Action
-> 000000fa2d1f1008 Microsoft.AspNet.SignalR.SqlServer.SqlReceiver
-> 000000fa2d1f1330 System.Action
-> 000000fa2d1f1308 System.Object[]
-> 000000fa2d1f12c8 System.Action
-> 000000fa2d1efb70 Microsoft.AspNet.SignalR.SqlServer.SqlStream
-> 000000fa2d1f1528 System.Action
-> 000000fa2d1f1500 System.Object[]
-> 000000fa2d1f14c0 System.Action
-> 000000fa2d1efb20 Microsoft.AspNet.SignalR.SqlServer.SqlMessageBus+<>c__DisplayClass3
-> 000000f92d0b84e0 Microsoft.AspNet.SignalR.SqlServer.SqlMessageBus
-> 000000f92d0b9568 System.Threading.Timer
-> 000000f92d0b96d8 System.Threading.TimerHolder
-> 000000f92d0b95a0 System.Threading.TimerQueueTimer
[... about 100 lines of the same TimerQueueTimer line above, but different memory addresses each time]
-> 000000f92cf1be68 System.Threading.TimerQueueTimer
-> 000000f92cf1be08 System.Threading.TimerCallback
-> 000000f92cf1bb48 System.Web.RequestTimeoutManager
-> 000000f92cf1bb80 System.Web.Util.DoubleLinkList[]
-> 000000f92cf1bc00 System.Web.Util.DoubleLinkList
-> 000000fb61323860 System.Web.RequestTimeoutManager+RequestTimeoutEntry
-> 000000fb6131fd38 System.Web.HttpContext
-> 000000fbe682a480 ASP.global_asax
-> 000000fbe682ac00 System.Web.HttpModuleCollection
-> 000000fbe682ac60 System.Collections.ArrayList
-> 000000fbe682b598 System.Object[]
-> 000000fbe682b018 System.Collections.Specialized.NameObjectCollectionBase+NameObjectEntry
-> 000000fbe682b000 System.Web.Routing.UrlRoutingModule
-> 000000faacec1f40 System.Web.Routing.RouteCollection
-> 000000faacec2030 System.Collections.Generic.List`1[[System.Web.Routing.RouteBase, System.Web]]
-> 000000fa2cfe4d80 System.Web.Routing.RouteBase[]
-> 000000f9acf14cd8 System.Web.Http.WebHost.Routing.HttpWebRoute
-> 000000f9acf149f8 System.Web.Http.Routing.RouteCollectionRoute
-> 000000f9acf1f4f0 System.Web.Http.Routing.SubRouteCollection
-> 000000f9acf1f510 System.Collections.Generic.List`1[[System.Web.Http.Routing.IHttpRoute, System.Web.Http]]
-> 000000fa2cf8f310 System.Web.Http.Routing.IHttpRoute[]
-> 000000fa2ceff770 System.Web.Http.Routing.HttpRoute
-> 000000fa2ceff678 System.Web.Http.Routing.HttpRouteValueDictionary
-> 000000fa2ceff6f0 System.Collections.Generic.Dictionary`2+Entry[[System.String, mscorlib],[System.Object, mscorlib]][]
-> 000000fa2cef9e78 System.Web.Http.Controllers.HttpActionDescriptor[]
-> 000000fa2cef7898 System.Web.Http.Controllers.ReflectedHttpActionDescriptor
-> 000000f9aced4608 System.Web.Http.HttpConfiguration
-> 000000f9aced4db0 System.Net.Http.Formatting.MediaTypeFormatterCollection
-> 000000f9aced6f40 System.Collections.Generic.List`1[[System.Net.Http.Formatting.MediaTypeFormatter, System.Net.Http.Formatting]]
-> 000000f9aced6f80 System.Net.Http.Formatting.MediaTypeFormatter[]
-> 000000f9aced4df8 System.Net.Http.Formatting.JsonMediaTypeFormatter
-> 000000f9acf1f448 System.Web.Http.Validation.ModelValidationRequiredMemberSelector
-> 000000f9acf1f468 System.Collections.Generic.List`1[[System.Web.Http.Validation.ModelValidatorProvider, System.Web.Http]]
-> 000000f9acf1f490 System.Web.Http.Validation.ModelValidatorProvider[]
-> 000000f9acf1db40 Ninject.Web.WebApi.Validation.NinjectDefaultModelValidatorProvider
-> 000000faaceca438 Ninject.StandardKernel
-> 000000faaceca498 Ninject.Components.ComponentContainer
-> 000000faaceca538 System.Collections.Generic.Dictionary`2[[System.Type, mscorlib],[Ninject.Components.INinjectComponent, Ninject]]
-> 000000f9acece000 System.Collections.Generic.Dictionary`2+Entry[[System.Type, mscorlib],[Ninject.Components.INinjectComponent, Ninject]][]
-> 000000f9acecdac8 Ninject.Activation.Caching.GarbageCollectionCachePruner
-> 000000f9acecdcb8 System.Threading.Timer
-> 000000f9acecdd30 System.Threading.TimerHolder
-> 000000f9acecdcd8 System.Threading.TimerQueueTimer
[... just under 1.5 million lines of the same TimerQueueTimer line above, but different memory addresses each time]
-> 000000f82dd4c028 System.Threading.TimerQueueTimer
-> 000000f82dd4bfc8 System.Threading.TimerCallback
-> 000000f82dd4ada0 System.Runtime.Caching.CacheExpires
-> 000000f82dd4add8 System.Runtime.Caching.ExpiresBucket[]
-> 000000f82dd4bc28 System.Runtime.Caching.ExpiresBucket
Running !objsize on the 000000faaceca438 Ninject.StandardKernel seems to take forever, implying that it references an awful lot of objects, possibly all 40 million of those System.Runtime.Caching.ExpiresBucket objects...
What is causing the leak? How should I go about identifying the offending class or code? There is no reference to any of our own code in the gcroot output, so is it due to a bug in one of the installed libraries we're using? Is it in Ninject? We're using v3.2.2 (not the latest, I know)
Posting as an answer because it's too long for a comment:
Seems to me Ninject is tracking an awful lot of scope-objects. AFAIR that's what the GarbageCollectionPruner is for. A scope object is something that is defined with .InScope(...this object here...) or with some of the overloads like InRequestScope().
GarbageCollectionPruner has a timer that periodically checks whether the scope is still "alive". And if it is not alive anymore (garbage collected) it will dispose & forget all objects associated with that scope.
Unless there's not a bug in ninject, this would mean your application is creating an awful lot of scopes in a short interval or they are not properly cleaned up (meaning: problem with your or other 3rd party code).
By the way, if the scope object implements INotifyWhenDisposed (ninject interface) the periodical IsAlive check is not necessary and also gives the benefit of "determnistic" disposal, i.E. just when the scope ends the scoped objects are disposed, too. Otherwise this is dependent on the GC and the timer in Ninject...
Related
I am using scatter-gatter in a Spring Integration flow to call 3 services parallel:
.scatterGather(rlr -> rlr.applySequence(true)
.recipientFlow(f1 -> f1
.channel(c -> c.executor(executorMLCalls))
.route(ifService1NeedsToBeCalled(), routeToService1OrBypassCall()))
.recipientFlow(f2 -> f2
.channel(c -> c.executor(executorMLCalls))
.enrich(this::service2RequestEnricher)
.enrich(this::service2Enricher))
.recipientFlow(f3 -> f3
.channel(c -> c.executor(executorMLCalls))
.enrich(this::service3RequestEnricher)
.enrich(this::service3Enricher)),
agg -> agg.correlationStrategy(msg -> msg.getHeaders().get("corrMLCalls"))
.releaseExpression("size() == 3"),
sgs -> sgs.gatherTimeout(gatherTimeout)
.requiresReply(true)
)
So the service2 and service3 have to be called always but calling the service1 depends on conditions.
Due to this I cannot use the size() == 3 release expression as sometimes it should be size() == 2.
How can I use dynamic value in the expression? Can I call a local method in the release expression?
Thank you!
Regards,
V.
Yes, such an expression is BeanFactory-aware and you can use a # syntax to reach any bean in your application context.
On the other hand there is a:
/**
* #param releaseStrategy the release strategy.
* #return the handler spec.
* #see AbstractCorrelatingMessageHandler#setReleaseStrategy(ReleaseStrategy)
*/
public S releaseStrategy(ReleaseStrategy releaseStrategy) {
Which may make any decission according your environment and MessageGroup as a releasing candidate.
This question already has answers here:
How do I write a correct micro-benchmark in Java?
(11 answers)
Closed 1 year ago.
I was learning java stream api and I found this issue in code.
Here is my code
long s = System.currentTimeMillis();
IntStream.range(1, 10).parallel().forEach( x ->{
System.out.println( "Thread -> "+ Thread.currentThread().getName() +" "+ x );
});
long e = System.currentTimeMillis();
System.out.println("Time taken "+ (e-s) + " ms");
System.out.println("---------------------");
s = System.currentTimeMillis();
IntStream.range(1, 10).forEach( x ->{
System.out.println( "Thread -> "+ Thread.currentThread().getName() +" "+ x );
});
e = System.currentTimeMillis();
System.out.println("Time taken "+ (e-s) + " ms");
Here is the output
Thread -> main 6
Thread -> ForkJoinPool.commonPool-worker-7 2
Thread -> ForkJoinPool.commonPool-worker-7 5
Thread -> ForkJoinPool.commonPool-worker-13 1
Thread -> ForkJoinPool.commonPool-worker-11 4
Thread -> ForkJoinPool.commonPool-worker-3 3
Thread -> ForkJoinPool.commonPool-worker-15 9
Thread -> ForkJoinPool.commonPool-worker-9 7
Thread -> ForkJoinPool.commonPool-worker-5 8
Time taken using parallel stream 29 ms
-------------------------------------
Thread -> main 1
Thread -> main 2
Thread -> main 3
Thread -> main 4
Thread -> main 5
Thread -> main 6
Thread -> main 7
Thread -> main 8
Thread -> main 9
Time taken simple stream 1 ms
I have tried this code on sts and inteliJ IDE, results are same. Parallel is taking longer time than sequential.
Is there a problem with my jdk? please suggest.
First of all, doing this kind of benchmarks is hard. All kinds of optimizations are performed during execution of the code. But in this case, I wouldn't expect a much different result when the benchmark would have been good.
First, parallellisation costs overhead. You need to start and stop threads, sometimes synchronize them at certain points, communicate between them, etc. So usually it is only worth the effort if there are a lot a elements to process, and the work which needs to be done for each element is non-trivial. Both are not true in your case.
Secondly, you are outputting to standard out. System.out will only allow one thread to output something at the same time. This is done, so that output isn't scrambled too much if multiple threads try to write to standard out. This means that in your case, all threads will basically spent a lot of time waiting for their turn to write.
So, no, there is nothing wrong with your JVM.
I am testing my erlang escript, the content of escriptis this:
#!/usr/bin/env escript
main([Action]) ->
case Action of
start -> start();
stop -> stop()
end;
main(_) ->
usage().
usage() ->
io:format("usage: factorial integer\n"),
halt(1).
start() -> io:format("Start.~n").
stop() -> io:format("Stop.~n").
But when I try to run the escript, I got this problem:
./escript start
escript: exception error: no case clause matching "start"
What caused this problem? Is the format of argument wrong?
Command line arguments are passed as strings to main/1, not atoms, so you need to match "start" and "stop", not start and stop:
...
main([Action]) ->
case Action of
"start" -> start();
"stop" -> stop()
end;
...
Unless your actual code is more complicated/different, you can also match ["start"] and ["stop"] in the function clause directly:
main(["start"]) -> start();
main(["stop"]) -> stop();
main(_) ->
usage().
This will call usage() for ./escript foo as well while your original script would crash, which may or may not be what you want.
Users of my program are organized hierarchically. Each user is a node in a tree, and the only other type of node is the department node. Each node has one and only one parent (possibly the root node).
I generated a DOT file to be used by the graphviz suite, but the resulting picture is unusable because it's too wide. I don't know if this program can be tuned to suit my needs, because I have a flat hierarchy with lots of sibling nodes, so maybe I need a program specifically designed for this (or write my own). I tried the unflatten tool, but without success.
This is the test dot file - note that my program often uses 10x this data...
Based on your comment, here's a quick test using other layout algorithms than dot. Since you said circle, I tried with circo and twopi.
I had to make some slight changes to the test script:
add some colors/shape/style/font
style user and department nodes differently
insert a newline between the name and the function of the users
Putting the long department names on two lines would probably help, too.
digraph G {
overlap=false;
splines=true;
root="node0";
node[colorscheme=paired12, fontsize=11];
node0 [shape=house, label="Organizzazione", style="filled", fillcolor=3, color="4", fontsize=20, height=2,
fontname="Times New Roman Bold"];
node[shape=doubleoctagon, style=filled, fillcolor=7, color=8, width=3];
node24 [label="C3 TERREMOTO"];
node28 [label="E POLIZIA URBANA E PROTEZIONE CIVILE"];
node14 [label="D1 SERVIZI SCOLASTICI"];
node35 [label="30: PROTOCOLLO ADMIN "];
node18 [label="B ECONOMICO E FINANZIARIO"];
node22 [label="C ASSETTO E TERRITORIO"];
node41 [label="A9 UFFICIO PUBBLICAZIONI"];
node38 [label="C1 TECNICO"];
node31 [label="A AFFARI GENERALI"];
node12 [label="A4 ANAGRAFE E STATO CIVILE"];
node20 [label="B1 TRIBUTI"];
node16 [label="A5 ELETTORALE E LEVA"];
node40 [label="31: PUBBLICAZIONI ALBO UTENTE "];
node26 [label="C2a TECNICO"];
node9 [label="A3 UFFICIO PROTOCOLLO CENTRALE"];
node[shape=box, style=filled, fillcolor="1", color="2", width=2];
node0 -> node24;
node0 -> node28;
node7 [label="14: ROCCO MARINACCIO\nVISUALIZZATORE"];
node0 -> node7;
node25 [label="27: FRANCESCO MARINO "];
node26 -> node25;
node39 [label="5: CIRO D'EMILIO\nPROTOCOLLATORE"];
node20 -> node39;
node4 [label="15: FRANCESCO PAZIENZA\nVISUALIZZATORE"];
node0 -> node4;
node3 [label="18: ADRIANA NATALE\nVISUALIZZATRICE"];
node0 -> node3;
node42 [label="4: MICHELE ROGATO\nRESP DI REPARTO"];
node18 -> node42;
node29 [label="29: FRANCESCO NOTA "];
node28 -> node29;
node0 -> node14;
node10 [label="12: STEFANO IEFFA "];
node0 -> node10;
node13 [label="20: ANTONIO MARINO "];
node14 -> node13;
node30 [label="3: PATRIZIA PLATANO\nPROTOCOLLATRICE"];
node31 -> node30;
node1 [label="19: PATRIZIA PLATANO "];
node0 -> node1;
node37 [label="6: GIUSEPPE CEGLIA\nRESP. DI REPARTO"];
node38 -> node37;
node0 -> node35;
node0 -> node18;
node23 [label="26: PASQUALE RUSSO "];
node24 -> node23;
node2 [label="17: MICHELE BICCARINO\nVISUALIZZATORE"];
node0 -> node2;
node0 -> node22;
node0 -> node41;
node11 [label="21: GIUSEPPE DI FLUMERI "];
node12 -> node11;
node43 [label="9: ROBERTO CAMPANELLA "];
node0 -> node43;
node0 -> node38;
node0 -> node31;
node0 -> node12;
node0 -> node20;
node17 [label="23: MICHELE IPPOLITO "];
node18 -> node17;
node6 [label="13: ALESSANDRO CAPANO "];
node0 -> node6;
node19 [label="24: SALVATORE DOTO "];
node20 -> node19;
node15 [label="22: BENVENUTA REA "];
node16 -> node15;
node27 [label="28: ANTONIO CAMPANELLA "];
node28 -> node27;
node8 [label="11: PASQUALE PALUMBO "];
node9 -> node8;
node5 [label="16: PAOLO PIETRO TROCCOLA\nVISUALIZZATORE"];
node0 -> node5;
node0 -> node16;
node21 [label="25: RAFFAELA COFANO "];
node22 -> node21;
node41 -> node40;
node32 [label="2: MARIA CRISTINA ANELLI\nVISUALIZZATRICE"];
node0 -> node32;
node0 -> node26;
node33 [label="10: VINCENZO BOTTICELLI "];
node0 -> node33;
node0 -> node9;
node36 [label="7: ANTONIETTA STRAZZELLA\nRESP. DI REPARTO"];
node22 -> node36;
node34 [label="1: AMBROGIO MASCIA\nAmministratore di sistema"];
node9 -> node34;
node44 [label="8: MARIANO LAUDISI\nVISUALIZZATORE"];
node0 -> node44;
}
You'll probably have to open the images and look at them in the original size.
Twopi layout:
Circo layout:
The circo layout could probably be more appropriate if you have department linked to departments, or a lot of users in one department.
The dot language has a lot of subtle ways to modify the image to make it more pleasing. Take a few minutes out and skim the dot guide. You can set constraints on pretty much any aspect from the X/Y ratio of the page, the repulsion and layout of nodes.
One of the contrib packages available with graphviz is GVEdit, which is a little Qt app that links to the graphviz libraries - one window to edit graphs, hit 'go' and it'll render to another window. Great instant feedback method of making things pretty.
What is the difference between each value of task_struct?
Here's my notes on the priority fields. I also include how data is presented via the 'ps' command (which gets it's data from /proc/pid/stat, among other things).
task_struct.prio:
0-99 -> Realtime
100-140 -> Normal priority
ps/stat "prio" field:
task_struct.prio - MAX_RT_PRIO (100)
(-100)-(-1) -> Realtime
0-40 -> Normal Priority
stat "rt_priority" field:
0 -> normal
1-99 -> realtime
stat "policy" field:
0 -> SCHED_OTHER (normal)
1 -> SCHED_FIFO
2 -> SCHED_RR (realtime)