public class Test{
public static void main(String[] args) {
Runtime runtime = Runtime.getRuntime();
System.out.println("total memory before "+runtime.totalMemory());
System.out.println("free memory before "+runtime.freeMemory());
for (int i=0;i < 1_000_000_000;i++){
Date date = new Date();
date=null;
}
System.out.println("total memory after "+runtime.totalMemory());
System.out.println("free memory after "+runtime.freeMemory());
System.gc();
System.out.println("total memory after gc "+runtime.totalMemory());
System.out.println("free memory after gc "+runtime.freeMemory());
}
}
I wrote the code based on the tutorial video. The result of my output is as follows.
This is the result on my computer
This is the result in the tutorial video.
1)Why after calling gc () the total memory and free memory are VERY reduced?
2). And why isn't this happening in the tutorial video.
Related
I use a Timer this way:
t = new Timer();
t.Interval = 10000;
t.Elapsed += ElapsedEvent;
and that's the ElapsedEvent method:
private void ElapsedEvent(object sender, ElapsedEventArgs e)
{
t.Stop();
try
{
var sessions = GetActiveSessions();
foreach (var session in sessions)
{
Task.Factory.StartNew(() => MyProcessTask(session));
}
}
catch (Exception ex)
{
}
t.Start();
}
i.e. it will be executed every 10000ms.
But the memory in RAM is increasing, ever, and I need to restart the Windows Service (where this code is executed).
Is the way I use Task.Factory incorrect? Does the memory allocated by thread never be released by Garbage collector this way?
Those parameters can be seen in a Task Manager.
The answer maybe a bit huge but it will answer the entire question except for the DISK USAGE part.
Getting the CPU utilization in percentage(%)
Not sure if WMI can be used for this case but i prefer not to use WMI because not all the machines support WMI.I will try to use other classes such as PerformanceCounter or so for the solutions.Here's a sample code that will return the CPU utilization in %
public FinalResult as float;
public async void GetCPUCounter()
{
CounterSample firstValue = cpuCounter.NextSample();
await Task.Delay(900); //delay is required because first call will always return 0
CounterSample secondValue = cpuCounter.NextSample();
FinalResult = CounterSample.Calculate(firstValue, secondValue);
await Task.Delay(900);
GetCPUCounter(); //calling again to get repeated values
}
Now, just use a Windows.Forms.Timer to repeatedly get the cpu utilization.Before that, just call GetCPUCounter() once from anywhere in your code, let's say from Form_load event :
private void Form_load(...)
{
GetCPUCounter();
}
private void mytimer_Tick(....)
{
string cpuUsage = finalresult.ToString() + "%"
}
Getting the memory utilization in percentage(%)
Here's a full class with 2 async methods that will return both memory usage and total memory:
using System;
using System.Diagnostics;
using System.Threading.Tasks;
using Microsoft.VisualBasic.Devices;
public class Memory
{
public int TotalRamInMb;
public int TotalRamInGb;
public double UsedRam;
public int UsedRamPercentage;
public string GetTotalRam()
{
var CI = new ComputerInfo();
var mem = ulong.Parse(CI.TotalPhysicalMemory.ToString());
int Mb = Convert.ToInt16(mem / (double)(1024 * 1024));
TotalRamInMb = Mb;
if (Mb.ToString().Length <= 3)
return Mb.ToString() + " MB physical memory";
else
{
return (Convert.ToInt16(Mb / (double)1024)).ToString() + " GB physical memory";
TotalRamInGb = Convert.ToInt16(Mb / (double)1024);
}
}
public async void GetUsedRam()
{
double URam;
Process[] allProc = Process.GetProcesses();
foreach (var proc in allProc)
URam += ((proc.PrivateMemorySize64) / (double)(1024 * 1024));
UsedRam = URam;
UsedRamPercentage = (UsedRam * 100) / TotalRamInMb;
await Task.Delay(900);
GetUsedRam();
}
}
Get the values from the variables such as UsedRam, UsedRamPercentage,TotalRamInMb and so... And make sure to call GetUsedRam() once like we did with GetCPUCounter and then use a Forms.Timer to repeatedly get the values from the above variables.
I will update the answer soon by adding Disk usage retrieval process . Cheers!
I have a two box cluster running Hazelcast 3.6.
I am trying to run some very basic tests to get a feel for performance. Gets, puts and an aggregation across a map of Integers are all seriously slow.
I am seeing about 2.6 get or put operations per second. Any idea why this might be?
This is my test code:
public static void main(String[] args) {
ClientConfig clientConfig = new ClientConfig();
clientConfig.getGroupConfig().setName("dev").setPassword("dev-pass");
clientConfig.getNetworkConfig().addAddress("10.90.288.14", "10.90.288.15");
HazelcastInstance client = HazelcastClient.newHazelcastClient(clientConfig);
IMap<Integer, Integer> map = client.getMap("int-map");
for (int i = 0; i < 10000; i++) {
map.put(i, i);
System.out.println(i);
}
for (int i = 0; i < 10000; i++) {
map.get(i);
System.out.println(i);
}
System.out.println("Summed:" + map.values().stream().mapToInt(Integer::intValue).sum());
client.shutdown();
System.out.println("Finished");
}
First thing I would fix is to remove the System.outs on every get since this has a significant overhead. If you really want to log something, add something like this:
if(k%1000==0)println("whatever")
Also you need a lot longer running time to say anything sensible, with 1k runs, you have not even heated up the JIT. So run at least 1M time.
And I would calculate the performance directly by storing the time before you start, and storing the time after the test. And you get something like
long duration=endMs-startMs;
double throughput = (1000d*iteration)/duration
I'm trying to understand one problem that I encountered recently in my project. I'm using Aurigma library to resize images. It is used in the single thread mode and produce only one thread during calculation. Lately I decided to move to ImageMagick project, because it is free and open source. I've built IM in the single thread mode and started to test. At first I wanted to compare their performance without interruptions, so I created a test that has high priorities for a process and its thread. Also, I set affinity to one core. I got that IM faster than Aurigma on ~25%. But than more threads I added than less IM had advantage against Aurigma.
My project is a windows service that starts about 7-10 child processes. Each process has 2 threads to process images. When I run my test as two different processes with 2 threads each, I noticed that IM worked worse than Aurigma on about 5%.
Maybe my question is not very detailed, but this scope is a little new for me and I would be glad to get direction for further investigation. How can it be that one program works faster if it is run on one thread in one process, but worse if it is run in multiple processes at the same time.
Fro example,
Au: 8 processes x 2Th(20 tasks per thread) = 320 tasks for 245 secs
IM: 8 processes x 2Th(20 tasks per thread) = 320 tasks for 280 secs
Au: 4 processes x 2Th(20 tasks per thread) = 160 tasks for 121 secs
IM: 4 processes x 2Th(20 tasks per thread) = 160 tasks for 141 secs
We can see that Au works better if we have more that 1 process, but in single process mode: Au process one task for 2,2 sec, IM for 1,4 sec and the sum time is better for IM
private static void ThreadRunner(
Action testFunc,
int repeatCount,
int threadCount
)
{
WaitHandle[] handles = new WaitHandle[threadCount];
var stopwatch = new Stopwatch();
// warmup
stopwatch.Start();
for (int h = 0; h < threadCount; h++)
{
var handle = new ManualResetEvent(false);
handles[h] = handle;
var thread = new Thread(() =>
{
Runner(testFunc, repeatCount);
handle.Set();
});
thread.Name = "Thread id" + h;
thread.IsBackground = true;
thread.Priority = ThreadPriority.Normal;
thread.Start();
}
WaitHandle.WaitAll(handles);
stopwatch.Stop();
Console.WriteLine("All Threads Total time taken " + stopwatch.ElapsedMilliseconds);
}
private static void Runner(
Action testFunc,
int count
)
{
//Process.GetCurrentProcess().ProcessorAffinity = new IntPtr(2); // Use only the second core
Process.GetCurrentProcess().PriorityClass = ProcessPriorityClass.BelowNormal;
Process.GetCurrentProcess().PriorityBoostEnabled = false;
Thread.CurrentThread.Priority = ThreadPriority.Normal;
var stopwatch = new Stopwatch();
// warmup
stopwatch.Start();
while(stopwatch.ElapsedMilliseconds < 10000)
testFunc();
stopwatch.Stop();
long elmsec = 0;
for (int i = 0; i < count; i++)
{
stopwatch.Reset();
stopwatch.Start();
testFunc();
stopwatch.Stop();
elmsec += stopwatch.ElapsedMilliseconds;
Console.WriteLine("Ticks: " + stopwatch.ElapsedTicks +
" mS: " + stopwatch.ElapsedMilliseconds + " Thread name: " + Thread.CurrentThread.Name);
}
Console.WriteLine("Total time taken " + elmsec + " Thread name: " + Thread.CurrentThread.Name);
}
/// <summary>
/// Entry point
/// </summary>
/// <param name="args"></param>
private static void Main(string[] args)
{
var files = GetFiles(args.FirstOrDefault());
if (!files.Any())
{
Console.WriteLine("Source files were not found.");
goto End;
}
//// Run tests
Console.WriteLine("ImageMagick run... Resize");
Runner(() => PerformanceTests.RunResizeImageMagickTest(files[0]), 20);
Console.WriteLine("Aurigma run... Resize");
Runner(() => PerformanceTests.RunResizeAurigmaTest(files[0]), 20);
Console.WriteLine("ImageMagick run... multi Resize");
ThreadRunner(() => PerformanceTests.RunResizeImageMagickTest(files[0]), 20, 2);
Console.WriteLine("Aurigma run... multi Resize");
ThreadRunner(() => PerformanceTests.RunResizeAurigmaTest(files[0]), 20, 2);
End:
Console.WriteLine("Done");
Console.ReadKey();
}
public static void RunResizeImageMagickTest(string source)
{
float[] ratios = { 0.25f, 0.8f, 1.4f };
// load the source bitmap
using (MagickImage bitmap = new MagickImage(source))
{
foreach (float ratio in ratios)
{
// determine the target image size
var size = new Size((int)Math.Round(bitmap.Width * ratio), (int)Math.Round(bitmap.Height * ratio));
MagickImage thumbnail = null;
try
{
thumbnail = new MagickImage(bitmap);
// scale the image down
thumbnail.Resize(size.Width, size.Height);
}
finally
{
if (thumbnail != null && thumbnail != bitmap)
{
thumbnail.Dispose();
}
}
}
}
}
public static void RunResizeAurigmaTest(string source)
{
float[] ratios = { 0.25f, 0.8f, 1.4f };
//// load the source bitmap
using (ABitmap bitmap = new ABitmap(source))
{
foreach (float ratio in ratios)
{
// determine the target image size
var size = new Size((int)Math.Round(bitmap.Width * ratio), (int)Math.Round(bitmap.Height * ratio));
ABitmap thumbnail = null;
try
{
thumbnail = new ABitmap();
// scale the image down
using (var resize = new Resize(size, InterpolationMode.HighQuality))
{
resize.ApplyTransform(bitmap, thumbnail);
}
}
finally
{
if (thumbnail != null && thumbnail != bitmap)
{
thumbnail.Dispose();
}
}
}
}
}
Code for testing is added. I use C#/.NET, ImageMagick works through ImageMagick.Net library, for Aurigma there is one too. For IM .net lib is written on C++/CLI, IM is C. A lot of languages are used.
OpenMP for IM is off.
Could be a memory cache issue. It is possible that multiple threads utilizing memory in a certain way create a scenario that one thread invalidates cache memory that another thread was using, causing a stall.
Programs that are not purely number crunching, but rely on a lot of IO (CPU<->Memory) are more difficult to analyze.
I am writing very large (both size and count) documents to a solr index(100s of fields with many numeric and some text fields) . I am using Tomcat 7 on W7 x64.
Based on #Maurico's suggestion when indexing millions of documents I parallelize the write operation (see code sample below)
The write to Solr method is being "Task"ed out from a main loop (Note: I task it out since the write op takes too long and holds up the main app)
The problem is that the memory consumption grows uncontrollably, the culprit is the solr write operations (when I comment them out the run works fine). How do I handle this issue? via Tomcat? or SolrNet?
Thanks for your suggestions.
//main loop:
{
:
:
:
//indexDocsList is the list I create in main loop and "chunk" it out to send to the task.
List<IndexDocument> indexDocsList = new List<IndexDocument>();
for(int n = 0; n< N; n++)
{
indexDocsList.Add(new IndexDocument{X=1, Y=2.....});
if(n%5==0) //every 5th time we write to solr
{
var chunk = new List<IndexDocument>(indexDocsList);
indexDocsList.Clear();
Task.Factory.StartNew(() => WriteToSolr(chunk)).ContinueWith(task => chunk.Clear());
GC.Collect();
}
}
}
private void WriteToSolr(List<IndexDocument> indexDocsList)
{
try
{
if (indexDocsList == null) return;
if (indexDocsList.Count <= 0) return;
int fromInclusive = 0;
int toExclusive = indexDocsList.Count;
int subRangeSize = 25;
//TO DO: This is still leaking some serious memory, need to fix this
ParallelLoopResult results = Parallel.ForEach(Partitioner.Create(fromInclusive, toExclusive, subRangeSize), (range) =>
{
_solr.AddRange(indexDocsList.GetRange(range.Item1, range.Item2 - range.Item1));
_solr.Commit();
});
indexDocsList.Clear();
GC.Collect();
}
catch (Exception ex)
{
logger.ErrorException("WriteToSolr()", ex);
}
finally
{
GC.Collect();
};
return;
}
You are manually committing after each batch. This is the most expensive operation for Solr. In your case, I would recommend autoCommit every x seconds and do a softAutoCommit (Solr 4.0) feature. That should take care of Solr's side of things. You'll also have to tweak your JVM garbage collection options so that you don't get pause the world GC.