It may be just a memory leak question. For those not familiar with WinSCP, skip ahead to question.
I am using .net assembly of WinSCP in a C++/CLI program. My program will read in a schedule file. The file instructs the program to transfer files from various locations. Some transfers could come from the same server, so my program should close the existing connection if it is a new server. If the server is the same, keep the connection to use.
As there is no Session::Close(), the document recommends to use Session::Dispose() (Refer to Session.Dispose() Documentation.) Yet when I compile, I see error message saying:
'Dispose' is not a member of 'WinSCP::Session'
Eventually I use delete session. Part of my program will then roughly look like:
void Transfer(String ^ sAction, String ^ sMode,
String ^ sSource_Server, String ^ sSource_Path,
String ^ sDest_Server, String ^ sDest_Path,
bool bDelDir, bool bDelFile )
{
if ((GlobalClass::g_sFtp_Server != sSource_Server && sAction == "G")
|| (GloblaClass::g_sFtp_Server != sDest_Server && sAction == "P"))
{
// Close existing connection first.
if (GlobalClass::g_sftpSession != nullptr)
delete GlobalClass::g_sftpSession;
if (GlobalClass::g_sftpSessionOptions != nullptr)
// Reuse the object
GlobalClass::g_sftpSessionOptions->HostName = sSource_Server;
else
{
// Recreate object and fill in detail
GlobalClass::g_sftpSessionOptions = gcnew WinSCP::SessionOptions();
GlobalClass::g_sftpSessionOptions->Protocol ....
GlobalClass::g_sftpSessionOptions->HostName ....
}
// Create new session
GlobalClass::g_sftpSession = gcnew WinSCP::Session();
GlobalClass::g_sftpSession->Open(GlobalClass::g_sftpSessionOptions);
// Set GlobalClass::g_sFtp_Server
}
// Transfer files accordingly...
}
【Question】: Will there be any memory leak by deleting the object (delete GlobalClass::g_sftpSession) and create it again (GlobalClass::g_sftpSession = gcnew WinSCP::Session()) many times per minute?
From several .net resources I have read, the delete object action will mark the object to be garbage collected. When will it be done? It is entirely up to the gc mechanism. So if my program has to make connections to several sites, it has to do that delete and create several times. By the program finishes (usually in less than 1 minute), can I count on the garbage collection mechanism to clean out all memory? Reason I ask is my program will run every minute. If there is memory leak each time program is run, my machine will be out of memory very soon.
The WinSCP .NET assembly Session class has the Dispose method. Though it's probably hidden by C++/CLI. You call the Dispose indirectly using the delete. See How do you dispose of an IDisposable in Managed C++ and Calling C++/CLI delete on C# object.
Generally, even if you do not, the garbage collector will do this for you (in an unpredictable moment), as you do not keep reference to old sessions. But it definitely won't let your machine run out of memory.
On the other hand, you NEED to call the Dispose (the delete) to close the unused sessions anyway, otherwise you may run out of allowed connections to the servers (or even exhaust servers' resources).
If you want to check, if and when the session is disposed, set the Session.DebugLogPath and search a log for an entry like:
[2014-04-23 08:08:50.756Z] [000a] Session.Dispose entering
Your question whether there's a chance for a memory leak, when a program finishes is irrelevant. Any memory allocated by a process is released by an operating system when the process finishes. No matter what leak/bug/anything is in the program itself. See also Does the heap get freed when the program exits? Anyway, I believe your code does not leak memory.
Related
I've got a "database lock" issue despite the sqlite has been compiled with -DSQLITE_THREADSAFE=2 and each thread uses its own database connection like this simplified code:
thread *t = new thread(RandomTextThread);
t.join();
and
static void RandomTextThread(void * aArg)
{
RandomText *rt1;
rt1 = new RandomText();
rt1->genRandomText();
}
and
RandomText::RandomText() {
int rc;
//
// open the database
//
rc = sqlite3_open("mydata.db.sqlite", &this->db);
}
So within each thread constructed by the RandomTextThread statical function, a new RandomText object will be created which in turn creates a new sqlite connection stored within the ->db class variable.
Sometimes the code will be execute DML commands. And this works for one thread without an error.
But two or more threads created by the mechanism above fails until as soon the threads will be make concurrent DML.
The system is old, I could compile sqlite 3.27 but I can use only GCC 4.2 with the TinyThread++ solution which in turn uses POSIX pthread.h .
Any help appreciate.
SQLITE_BUSY is an expected error code when a database is both read and modified at the same time. Using Write-Ahead Logging, it is possibly to greatly reduce the number of occurrences if there is just one writer and many readers.
sqlite3_exec may keep a lock after completion if you have disabled auto-commit mode, for example using the BEGIN command. In this case, you must issue a COMMIT or ROLLBACK command to release the lock.
You can use sqlite3_busy_timeout to install a buys handler which performs waiting inside SQLite itself.
As the title says, how do two or more threads share memory on the heap that they have allocated? I've been thinking about it and I can't figure out how they can do it. Here is my understanding of the process, presumably I am wrong somewhere.
Any thread can add or remove a given number of bytes on the heap by making a system call which returns a pointer to this data, presumably by writing to a register which the thread can then copy to the stack.
So two threads A and B can allocate as much memory as they want. But I don't see how thread A could know where the memory that thread B has allocated is located. Nor do I know how either thread could know where the other thread's stack is located. Multi-threaded programs share the heap and, I believe, can access one another's stack but I can't figure out how.
I tried searching for this question but only found language specific versions that abstract away the details.
Edit:
I am trying not to be language or OS specific but I am using Linux and am looking at it from a low level perspective, assembly I guess.
My interpretation of your question: How can thread A get to know a pointer to the memory B is using? How can they exchange data?
Answer: They usually start with a common pointer to a common memory area. That allows them to exchange other data including pointers to other data with each other.
Example:
Main thread allocates some shared memory and stores its location in p
Main thread starts two worker threads, passing the pointer p to them
The workers can now use p and work on the data pointed to by p
And in a real language (C#) it looks like this:
//start function ThreadProc and pass someData to it
new Thread(ThreadProc).Start(someData)
Threads usually do not access each others stack. Everything starts from one pointer passed to the thread procedure.
Creating a thread is an OS function. It works like this:
The application calls the OS using the standard ABI/API
The OS allocates stack memory and internal data structures
The OS "forges" the first stack frame: It sets the instruction pointer to ThreadProc and "pushes" someData onto the stack. I say "forge" because this first stack frame does not arise naturally but is created by the OS artificially.
The OS schedules the thread. ThreadProc does not know it has been setup on a fresh stack. All it knows is that someData is at the usual stack position where it would expect it.
And that is how someData arrives in ThreadProc. This is the way the first, initial data item is shared. Steps 1-3 are executed synchronously by the parent thread. 4 happens on the child thread.
A really short answer from a bird's view (1000 miles above):
Threads are execution paths of the same process, and the heap actually belongs to the process (and as a result shared by the threads). Each threads just needs its own stack to function as a separate unit of work.
Threads can share memory on a heap if they both use the same heap. By default most languages/frameworks have a single default heap that code can use to allocate memory from the heap. In unmanaged languages you generally make explicit calls to allocate heap memory. In C, that might be malloc, etc. for example. In managed languages heap allocation is usually automatic and how allocation is done depends on the language--usually through the use of the new operator. but, that depends slightly on context. If you provide the OS or language context you're asking about, I might be able to provide more detail.
A Thread shared with other threads belonging to the same process: its code section, data section and other operating system resources such as open files and signals.
The part you are missing is static memory containing static variables.
This memory is allocated when the program is started, and assigned known adresses (determined at the linking time). All threads can access this memory without exchanging any data runtime, because the addresses are effectively hardcoded.
A simple example might look like this:
// Global variable.
std::atomic<int> common_var;
void thread1() {
common_var = compute_some_value();
}
void thread2() {
do_something();
int current_value = common_var;
do_more();
}
And of course the global value may be a pointer, that can be used to exchange heap memory. The producer allocates some objects, the consumer takes and uses them.
// Global variable.
std::atomic<bool> produced;
SomeData* data_pointer;
void producer_thread() {
while (true) {
if (!produced) {
SomeData* new_data = new SomeData();
data_pointer = new_data;
// Let the other thread know there is something to read.
produced = true;
}
}
}
void consumer_thread() {
while (true) {
if (produced) {
SomeData* my_data = data_pointer;
data_pointer = nullptr;
// Let the other thread know we took the data.
produced = false;
do_something_with(my_data);
delete my_data;
}
}
}
Please note: these are not examples of good concurrent code, but they show the general idea without too much clutter.
I have a Qt object that's used by a GUI thread and a networking thread. It looks like:
QString User::Username()
{
QMutexLocker locker(&mutex);
return username;
}
void User::SetUsername(const QString &newUsername)
{
QMutexLocker locker(&mutex);
username = newUsername;
}
QString User::Password()
{
QMutexLocker locker(&mutex);
return password;
}
...
Both the GUI and networking thread may use the object (e.g. to display the username on the screen, and to get the username to send across the network).
I'm worried something is wrong, as every method in the object has a QMutexLocker line, to make it thread safe.
Is it acceptable to use QMutexLocker in this way, or is the code structured badly?
You should be using QReadWriteLock and QReadLocker or QWriteLocker respectively. So no threads will be locked if there are only reading threads.
If there are some fields of the class which are accessed changed very frequently, and which dont change any other state of the class, you might want to give it its own dedicated lock.
I think you may be going about things the wrong way. Serializing each method call will "sort of" work, but it won't reliably handle operations like adding or removing a User object. For example, if your main thread deletes the User object, it won't matter that the network thread is carefully locking a mutex, because after the mutex-lock operation returns, the network thread will then try to access the (now deleted) User object, and trying to read OR write freed memory will cause your app to crash (or worse, just mysteriously do the wrong thing sometimes).
Here's a better way to do it (assuming that the User objects are reasonably small): Instead of having the network thread and the I/O thread share the same User object, and trying to serialize all accesses to the object at the method level, you'd be better off giving a separate copy of each User object to the I/O thread. Then when one thread changes its local copy of the User object, it should send a message to the other thread containing a copy of the updated object, and when the other thread receives the message it can update its local copy to match again. That way each thread has exclusive read/write access to its own local set of User objects, and can read/write them without any locking. This also allows each thread to add or remove objects at will (as long as it sends an update-message to the other thread afterwards, so the other thread will follow suit).
I think a better and cleaner way would be to have a "safe section"
updateUser( User ) {
User.acquireLock()
User.SetUsername(newUsername)
User.Password()
< more operations here >
User.releaseLock()
}
The advantages of this is that you are locking only once the mutex( that is an expensive operation).
Is there any way to determine if an object is locked in C#? I have the unenviable position, through design where I'm reading from a queue inside a class, and I need to dump the contents into a collection in the class. But that collection is also read/write from an interface outside the class. So obviously there may be a case when the collection is being written to, as the same time I want to write to it.
I could program round it, say using delegate but it would be ugly.
You can always call the static TryEnter method on the Monitor class using a value of 0 for the value to wait. If it is locked, then the call will return false.
However, the problem here is that you need to make sure that the list that you are trying to synchronize access to is being locked on itself in order to synchronize access.
It's generally bad practice to use the object that access is being synchronized as the object to lock on (exposing too much of the internal details of an object).
Remember, the lock could be on anything else, so just calling this on that list is pointless unless you are sure that list is what is being locked on.
Monitor.TryEnter will succeed if the object isn't locked, and will return false if at this very moment, the object is locked. However, note that there's an implicit race here: the instance this method returns, the object may not be locked any more.
I'm not sure if a static call to TryEnter with a time of 0 will guarantee that the lock will not be acquired if it is available. The solution I did to test in debug mode that the sync variable was locked was using the following:
#if DEBUG
// Make sure we're inside a lock of the SyncRoot by trying to lock it.
// If we're able to lock it, that means that it wasn't locked in the first
// place. Afterwards, we release the lock if we had obtained it.
bool acquired = false;
try
{
acquired = Monitor.TryEnter(SyncRoot);
}
finally
{
if (acquired)
{
Monitor.Exit(SyncRoot);
}
}
Debug.Assert(acquired == false, "The SyncRoot is not locked.");
#endif
Monitor.IsEntered
Determines whether the current thread holds the lock on the specified object.
Available since 4.5
Currently you may call Monitor.TryEnter to inspect whether object is locked or not.
In .NET 4.0 CLR team is going to add "Lock inspection API"
Here is a quotation from Rick Byers article:
lock inspection
We're adding some simple APIs to ICorDebug which allow you to explore managed locks (Monitors). For example, if a thread is blocked waiting for a lock, you can find what other thread is currently holding the lock (and if there is a time-out).
So, with this API you will be able to check:
1) What object is holding a lock?
2) Who’s waiting for it?
Hope this helps.
I remembered someone telling me one good one. But i cannot remember it. I spent the last 20mins with google trying to learn more.
What are examples of bad/not great code that causes a performance hit due to garbage collection ?
from an old sun tech tip -- sometimes it helps to explicitly nullify references in order to make them eligible for garbage collection earlier:
public class Stack {
private static final int MAXLEN = 10;
private Object stk[] = new Object[MAXLEN];
private int stkp = -1;
public void push(Object p) {stk[++stkp] = p;}
public Object pop() {return stk[stkp--];}
}
rewriting the pop method in this way helps ensure that garbage collection gets done in a timely fashion:
public Object pop() {
Object p = stk[stkp];
stk[stkp--] = null;
return p;
}
What are examples of bad/not great code that causes a performance hit due to garbage collection ?
The following will be inefficient when using a generational garbage collector:
Mutating references in the heap because write barriers are significantly more expensive than pointer writes. Consider replacing heap allocation and references with an array of value types and an integer index into the array, respectively.
Creating long-lived temporaries. When they survive the nursery generation they must be marked, copied and all pointers to them updated. If it is possible to coalesce updates in order to reuse of an old version of a collection, do so.
Complicated heap topologies. Again, consider replacing many references with indices.
Deep thread stacks. Try to keep stacks shallow to make it easier for the GC to collate the global roots.
However, I would not call these "bad" because there is nothing objectively wrong with them. They are only inefficient when used with this kind of garbage collector. With manual memory management, none of the issues arise (although many are replaced with equivalent issues, e.g. performance of malloc vs pool allocators). With other kinds of GC some of these issues disappear, e.g. some GCs don't have a write barrier, mark-region GCs should handle long-lived temporaries better, not all VMs need thread stacks.
When you have some loop involving the creation of new object's instances: if the number of cycles is very high you procuce a lot of trash causing the Garbage Collector to run more frequently and so decreasing performance.
One example would be object references that are kept in member variables oder static variables. Here is an example:
class Something {
static HugeInstance instance = new HugeInstance();
}
The problem is the garbage collector has no way of knowing, when this instance is not needed anymore. So its usually better to keep things in local variables and have small functions.
String foo = new String("a" + "b" + "c");
I understand Java is better about this now, but in the early days that would involve the creation and destruction of 3 or 4 string objects.
I can give you an example that will work with the .Net CLR GC:
If you override a finalize method from a class and do not call the super class Finalize method such as
protected override void Finalize(){
Console.WriteLine("Im done");
//base.Finalize(); => you should call him!!!!!
}
When you resurrect an object by accident
protected override void Finalize(){
Application.ObjJolder = this;
}
class Application{
static public object ObjHolder;
}
When you use an object that uses Finalize it takes two GC collections to get rid of the data, and in any of the above codes you won't delete it.
frequent memory allocations
lack of memory reusing (when dealing with large memory chunks)
keeping objects longer than needed (keeping references on obsolete objects)
In most modern collectors, any use of finalization will slow the collector down. And not just for the objects that have finalizers.
Your custom service does not have a load limiter on it, so:
A lot requests come in for some reason at the same time (everyone logs on in the morning say)
The service takes longer to process each requests as it now has 100s of threads (1 per request)
Yet more part processed requests builds up due to the longer processing time.
Each part processed request has created lots of objects that live until the end of processing that request.
The garbage collector spends lots of time trying to free memory it, however it can’t due to the above.
Yet more part processed requests builds up due to the longer processing time…. (including time in GC)
I have encountered a nice example while doing some parallel cell based simulation in Python. Cells are initialized and sent to worker processes after pickling for running. If you have too many cells at any one time the master node runs out of ram. The trick is to make a limited number of cells pack them and send them off to cluster nodes before making some more, remember to set the objects already sent off to "None". This allows you to perform large simulations using the total RAM of the cluster in addition to the computing power.
The application here was cell based fire simulation, only the cells actively burning were kept as objects at any one time.