I have been asked to debug, and improve, a complex multithreaded app, written by someone I don't have access to, that uses concurrent queues (both GCD and NSOperationQueue). I don't have access to a plan of the multithreaded architecture, that's to say a high-level design document of what is supposed to happen when. I need to create such a plan in order to understand how the app works and what it's doing.
When running the code and debugging, I can see in Xcode's Debug Navigator the various threads that are running. Is there a way of identifying where in the source-code a particular thread was spawned? And is there a way of determining to which NSOperationQueue an NSOperation belongs?
For example, I can see in the Debug Navigator (or by using LLDB's "thread backtrace" command) a thread's stacktrace, but the 'earliest' user code I can view is the overridden (NSOperation*) start method - stepping back earlier in the stack than that just shows the assembly instructions for the framework that invokes that method (e.g. __block_global_6, _dispatch_call_block_and_release and so on).
I've investigated and sought various debugging methods but without success. The nearest I got was the idea of method swizzling, but I don't think that's going to work for, say, queued NSOperation threads. Forgive my vagueness please: I'm aware that having looked as hard as I have, I'm probably asking the wrong question, and probably therefore haven't formed the question quite clearly in my own mind, but I'm asking the community for help!
Thanks
The best I can think of is to put breakpoints on dispatch_async, -[NSOperation init], -[NSOperationQueue addOperation:] and so on. You could configure those breakpoints to log their stacktrace, possibly some other info (like the block's address for dispatch_async, or the address of the queue and operation for addOperation:), and then continue running. You could then look though the logs when you're curious where a particular block came from and see what was invoked and from where. (It would still take some detective work.)
You could also accomplish something similar with dtrace if the breakpoints method is too slow.
Related
I'm writing a command line tool for installing Windows services using Node JS. After running a bunch of async operations, my tool should print a success message then quit. Sometimes however, it prints its success message and doesn't quit.
Is there a way to view what is queued on Node's internal event loop, so I can see what is preventing my tool from quitting?
The most typical culprit for me in CLI apps is event listeners that are keeping the process alive. I obviously can't say if that's relevant to you without seeing your code, though.
To answer your more general question, I don't believe there are any direct ways to view all outstanding tasks in the event loop (at least not from JS-land). You can, however, get pretty close with process._getActiveHandles() and process._getActiveRequests().
I really recommend you look up the documentation for them, though. Because you won't find any. They're undocumented. And they start with underscores. Use at your own peril. :)
try to use some tools to clarify the workflow - for example, the https://github.com/caolan/async#waterfall or https://github.com/caolan/async#eachseriesarr-iterator-callback
so, you don't lose the callback called and can catch any erros thrown while executing commands.
I think you also need to provide some code samples that leads to this errors.
Edit : If you're seeing this same problem (and you're accustomed to NOT seeing this under VS2010) please comment below so I know it's not just me - but be sure to check Han's answer to make sure none of those scenarios appear...
I've been updating my app to run with .NET 4.5 in VS2012 RTM and noticing something that I don't quite understand and that is unexpectedly green highlighted statements (instead of yellow).
Now I'm well aware of what this is supposed to mean, and the IDE is even showing me a little explanation tooltip.
This is the next statement to execute when this thread returns from
the current function
However there's absolutely nothing asynchronous or thread based about this code. In this simple example I'm sure you'll agree that string.ToUpper() won't be off in another thread. I can step through the code no issue.
There's nothing else going on and I am on the main thread as you can see here.
I am using async and await and MVVM-Light (the above method is the result of a RelayCommand) but I still get this behavior even when the code path is directly off an event handler such as PreviewKeyDown.
If I create a new app I cannot duplicate this - the coloring is yellow as expected - even when using await.
Anybody got any idea? It's starting to drive me crazy!!
It is green when the current instruction pointer is not exactly at the start of the statement. Some common causes:
Common in threaded code, setting a breakpoint in one thread and switching context to another. The other thread will have been interrupted by the debugger at an entirely random location. Often in code that you don't have source code or debugging info for, like String.ToUpper(), the debugger can only show the "closest" source code
Using Debugger + Break All to break into the debugger. Same idea as above, the instruction pointer will be at a random address
Getting an exception in code you don't have debugging info for. The editor shows the last entry in the Call Stack that it does have source code for. You need the call stack window to see where the actual exception was raised. Or the Exception Assistant, its reason for being
Debugging optimized code. The jitter optimizer scrambles the code pretty heavily, making it likely that the debugger can't show the current location accurately
Having out-dated debugging info or editing the code while debugging
Debugging code generated by the x64 jitter, happens when the project's Target Platform setting is AnyCPU. The x64 jitter has a number of chronic bugs that are not getting fixed, generating incorrect debugging info is one of them. Problems that were not addressed until it was completely rewritten, done by the RyuJIT project and first available in .NET version 4.6. Targeting x86 in your EXE project is the workaround.
I understand that this is old post yet I would like to answer the question with my experience.
I have encountered same issue recently in one of my WCF application. After debugging and closely looking service logs and I find out that my code was giving this error because service was hitting max allowed limit for code execution and once the service hit max allowed time limit it was trying to offload the current debugging session.
ERROR IN GREEN STATEMENT: this is the next statement to execute when thread return
So avoiding this issue you can try to look any potential code(Code/Service Timeout or any other code block) which is trying to offload your currently executing code context and try to fix it, furthermore original explanation given by #Hans is still very much relevant for trouble shooting this issue.
Actually, I am also facing this issue. This is because I missed some layout component in landscape mode, So check all the Id's and components and Run, you will not get this error.
I've been learning some lua for game development. I heard about coroutines in other languages but really came up on them in lua. I just don't really understand how useful they are, I heard a lot of talk how it can be a way to do multi-threaded things but aren't they run in order? So what benefit would there be from normal functions that also run in order? I'm just not getting how different they are from functions except that they can pause and let another run for a second. Seems like the use case scenarios wouldn't be that huge to me.
Anyone care to shed some light as to why someone would benefit from them?
Especially insight from a game programming perspective would be nice^^
OK, think in terms of game development.
Let's say you're doing a cutscene or perhaps a tutorial. Either way, what you have are an ordered sequence of commands sent to some number of entities. An entity moves to a location, talks to a guy, then walks elsewhere. And so forth. Some commands cannot start until others have finished.
Now look back at how your game works. Every frame, it must process AI, collision tests, animation, rendering, and sound, among possibly other things. You can only think every frame. So how do you put this kind of code in, where you have to wait for some action to complete before doing the next one?
If you built a system in C++, what you would have is something that ran before the AI. It would have a sequence of commands to process. Some of those commands would be instantaneous, like "tell entity X to go here" or "spawn entity Y here." Others would have to wait, such as "tell entity Z to go here and don't process anymore commands until it has gone here." The command processor would have to be called every frame, and it would have to understand complex conditions like "entity is at location" and so forth.
In Lua, it would look like this:
local entityX = game:GetEntity("entityX");
entityX:GoToLocation(locX);
local entityY = game:SpawnEntity("entityY", locY);
local entityZ = game:GetEntity("entityZ");
entityZ:GoToLocation(locZ);
do
coroutine.yield();
until (entityZ:isAtLocation(locZ));
return;
On the C++ size, you would resume this script once per frame until it is done. Once it returns, you know that the cutscene is over, so you can return control to the user.
Look at how simple that Lua logic is. It does exactly what it says it does. It's clear, obvious, and therefore very difficult to get wrong.
The power of coroutines is in being able to partially accomplish some task, wait for a condition to become true, then move on to the next task.
Coroutines in a game:
Easy to use, Easy to screw up when used in many places.
Just be careful and not use it in many places.
Don't make your Entire AI code dependent on Coroutines.
Coroutines are good for making a quick fix when a state is introduced which did not exist before.
This is exactly what java does. Sleep() and Wait()
Both functions are the best ways to make it impossible to debug your game.
If I were you I would completely avoid any code which has to use a Wait() function like a Coroutine does.
OpenGL API is something you should take note of. It never uses a wait() function but instead uses a clean state machine which knows exactly what state what object is at.
If you use coroutines you end with up so many stateless pieces of code that it most surely will be overwhelming to debug.
Coroutines are good when you are making an application like Text Editor ..bank application .. server ..database etc (not a game).
Bad when you are making a game where anything can happen at any point of time, you need to have states.
So, in my view coroutines are a bad way of programming and a excuse to write small stateless code.
But that's just me.
It's more like a religion. Some people believe in coroutines, some don't. The usecase, the implementation and the environment all together will result into a benefit or not.
Don't trust benchmarks which try to proof that coroutines on a multicore cpu are faster than a loop in a single thread: it would be a shame if it were slower!
If this runs later on some hardware where all cores are always under load, it will turn out to be slower - ups...
So there is no benefit per se.
Sometimes it's convenient to use. But if you end up with tons of coroutines yielding and states that went out of scope you'll curse coroutines. But at least it isn't the coroutines framework, it's still you.
We use them on a project I am working on. The main benefit for us is that sometimes with asynchronous code, there are points where it is important that certain parts are run in order because of some dependencies. If you use coroutines, you can force one process to wait for another process to complete. They aren't the only way to do this, but they can be a lot simpler than some other methods.
I'm just not getting how different they are from functions except that
they can pause and let another run for a second.
That's a pretty important property. I worked on a game engine which used them for timing. For example, we had an engine that ran at 10 ticks a second, and you could WaitTicks(x) to wait x number of ticks, and in the user layer, you could run WaitFrames(x) to wait x frames.
Even professional native concurrency libraries use the same kind of yielding behaviour.
Lots of good examples for game developers. I'll give another in the application extension space. Consider the scenario where the application has an engine that can run a users routines in Lua while doing the core functionality in C. If the user needs to wait for the engine to get to a specific state (e.g. waiting for data to be received), you either have to:
multi-thread the C program to run Lua in a separate thread and add in locking and synchronization methods,
abend the Lua routine and retry from the beginning with a state passed to the function to skip anything, least you rerun some code that should only be run once, or
yield the Lua routine and resume it once the state has been reached in C
The third option is the easiest for me to implement, avoiding the need to handle multi-threading on multiple platforms. It also allows the user's code to run unmodified, appearing as if the function they called took a long time.
I must say I know a lot of things about threads, but theres something that is driving me crazy.
I use to wait for a thread using the Windows API function WaitForSingleObject and it works fine. But, when i am debugging my code, it seems that WaitForSingleObject is very, very, very slow (it hangs a lot). But when I am just running my app without debug, WaitForSingleObject is very, very fast.
Why does this happen? Is it because of Messages that the IDE sends? Or is it because of the compiler?
This is not affecting me a lot. I just think this issue is really annoying.
Edit: I am using Delphi 2010.
I really doubt that WaitForSingleObject is in fact the ONLY thing that gets slower. Rather, it is probable that almost everything gets slower, when you run with debugging on.
I find that far more than Win32 API calls, calls to OutputdebugString slow me down, and anything that the IDE chooses to log in the event view, really, because a big load of these event or output messages, slows the IDE, the debugger, and thus the program that I'm debugging, a lot.
Try turning off the event view checkboxes in the configuration menu and see if EVERYTHING gets faster.
It's in Tools -> Options, as shown by the OP in his image, which I've also added here, for handy access:
Add more ram :) you probably have a not so powerful machine. I had a similar problem but using vs2010. Anyway the principles are the same: Running in debug mode adds an overhead from the checks added by compiler to the code and move overhead from the environment itself that must manages threads, resources and stuff for the case you want to see them.
PS: What makes you think the problem are the Wait* functions ? Have you made a simple application that use Wait* and behaves the same ?
What's a good way to leverage TDD to drive out thread-safe code? For example, say I have a factory method that utilizes lazy initialization to create only one instance of a class, and return it thereafter:
private TextLineEncoder textLineEncoder;
...
public ProtocolEncoder getEncoder() throws Exception {
if(textLineEncoder == null)
textLineEncoder = new TextLineEncoder();
return textLineEncoder;
}
Now, I want to write a test in good TDD fashion that forces me to make this code thread-safe. Specifically, when two threads call this method at the same time, I don't want to create two instances and discard one. This is easily done, but how can I write a test that makes me do it?
I'm asking this in Java, but the answer should be more broadly applicable.
You could inject a "provider" (a really simple factory) that is responsible for just this line:
textLineEncoder = new TextLineEncoder();
Then your test would inject a really slow implementation of the provider. That way the two threads in the test could more easily collide. You could go as far as have the first thread wait on a Semaphore that would be released by the second thread. Then success of the test would ensure that the waiting thread times out. By giving the first thread a head-start you can make sure that it's waiting before the second one releases.
It's hard, though possible - possibly harder than it's worth. Known solutions involve instrumenting the code under test. The discussion here, "Extreme Programming Challenge Fourteen" is worth sifting through.
In the book Clean Code there are some tips on how to test concurrent code. One tip that has helped me to find concurrency bugs, is running concurrently more tests than the CPU has cores.
In my project, running the tests takes about 2 seconds on my quad core machine. When I want to test the concurrent parts (there are some tests for that), I hold down in IntelliJ IDEA the hotkey for running all tests, until I see in the status bar that 20, 50 or 100 test runs are in execution. I follow in Windows Task Manager the CPU and memory usage, to find out when all the test runs have finished executing (memory usage goes up by 1-2 GB when they all are running and then slowly goes back down).
Then I close one by one all the test run output dialogs, and check that there were no failures. Sometimes there are failed tests or tests which are in deadlock, and then I investigate them until I find the bug and have fixed it. That has helped me to find a couple of nasty concurrency bugs. The most important thing, when facing an exception/deadlock that should not have happened, is always assuming that the code is broken, and investigating the reason ruthlessly and fixing it. There are no cosmic rays which cause programs to crash randomly - bugs in code cause programs to crash.
There are also frameworks such as http://www.alphaworks.ibm.com/tech/contest which use bytecode manipulation to force the code to do more thread switching, thus increasing the probability of making concurrency bugs visible.
When I test drove an implementation that needed to be thread safe recently I came up with the solution I provided as an answer for this question. Hope that helps even though there are no tests there. Hope link is OK raher than duplicating teh answer...
Chapter 12 of Java Concurrency in Practice is called "Testing Concurrent Programs". It documents testing for safety and liveness, but says this is a hard subject. I am not sure this problem is solvable by the tools of that chapter.
Just off the top of my head could you compare the instances returned to see if they are indeed the same instance or if they are different? That's probably where I would start with C#, I would imagine you can do the same in java