Related
In POSIX, because of "spurious wakeup" problem, programmers are forced to use while() instead of if when checking condition.
I think spurious wakeup is unintuitive and confusing problem, but I thought it's an inevitable problem.
Recently, I found that event objects of win32 doesn't have "spurious wakeup" problem.
Why POSIX system and other systems still use condition variable which has "spurious wakeup" problem? (despite this can be solved?)
You ask:
Why POSIX system and other systems still use condition variable which has "spurious wakeup" problem? (despite this can be solved?)
Basically, it's faster than the alternative.
The RATIONALE section of the POSIX.1-2017 treatment of pthread_cond_broadcast and pthread_cond_signal specifically has this to say about "Multiple Awakenings by Condition Signal":
While [the "spurious wakeup"] problem could be resolved, the loss of efficiency for a fringe condition that occurs only rarely is unacceptable, especially given that one has to check the predicate associated with a condition variable anyway. Correcting this problem would unnecessarily reduce the degree of concurrency in this basic building block for all higher-level synchronization operations.
Emphasis added.
The text further observes that forcing "a predicate-testing-loop around the condition wait" is a more robust coding practice than the alternative, because the application will necessarily tolerate superfluous broadcasts and signals from elsewhere in the code.
I don't study this area of computing to be honest. Actually my references is some web and academic articles then I'm insecure but curious about some concepts of parallel computing.
I've formulated two sentences and would like to validate it.
First one:
Imperative languages uses variables to abstract hardware memory of a computer. If there is two parallel threads and at least one performs a write, without synchronization mechanisms, a data race happens.
We can consider that data races are intrinsic to Imperative Programming paradigm?
Second one:
Data races can produces unexpected results. Since data races occurs with multiple parallel threads, then they are a problem of multithreading ability.
We can consider that data races is an obstacle not only specifically to multithreading but for parallel computing in general?
My real goal is write some text correlating Imperative Programming and Parallel Processing to explain the benefits of Functional Programming. Any correction and further information is welcome.
Data races are about event chronology, and aren't even particular to statements in code, consider the following (single-threaded) JavaScript:
let fs = require('fs');
let dirContents = fs.readdirSync('./');
let files = dirContents.map(fname => fs.readFileSync(fname, 'utf-8'));
This code contains a data race, not because of anything being done by the code itself, but because some other program could have come along and deleted the last file in the list between the time we (sequentially) started reading the first file and the time we try to read that last one. The code is deceptively short but there's an imperative iteration still occuring under the hood in both lines 2 and 3.
And JavaScript despite being single-threaded is full of concurrency, the reason the above fs functions have 'Sync' in the name is because the default versions are asynchronous and its surprisingly easy to introduce race conditions in the code itself in JavaScript. So threads are a red herring, they just make it arguably more likely that data races exist in the code, their absence proves nothing.
There are only two* ways to work around this problem. One is to use resource locks to attempt to ensure that events happen in an expected order. Another is to ensure that nothing ever changes in place. And that is the functional programming approach: concepts like immutable data, the whole program is one big expression rather than a series of steps, etc.
*There is actually another way, you can construct a formal proof of the correctness of the program. This method is somewhat impractical, at least at this particular moment in the history of computer science.
I am new to concurrent programming. As I was going through the topics I got confused between Synchronizations, thread safe collections, atomic wrapper classes, locks.
Locks and Synchronization do same work by making a piece of code thread safe. Why do we need thread safe collections or atomic wrapper classes then? As locking will enable only a single thread to access the code and won't let collections or primitive types to be thread unsafe.
This is a very wide question that you're asking. Problem is, not all of these things have a single, strict definition. For example, thread-safe collections might use various forms of synchronization (like e.g. locks or atomic operations) to achieve thread-safety. However, not even the term "thread-safe" is well-defined!
However, there is one thing you got wrong surely: Synchronization is the goal, while locks, mutexes, atomics etc are means to achieve that. Synchronization just means that different threads access resources in a synchronized way. In other words, they coordinate access so that they don't badly interact with each other. BTW: I'm talking here about threads, but the different entities could also be processes or even different computers, but let's keep it simple at first.
Now, you ask about the use of "thread safe collections or atomic wrapper classes" and why they are required at all. The answer is pretty simple, these things provide different interfaces or services on a higher level. For example, when I have a FIFO connecting two threads, it doesn't matter how they synchronize access to the underlying queue. If the interface for the two threads is implemented properly, you get certain guarantees. Doing so manually with just locks is possible but complicated, so providing these as high-level building blocks in addition to the low-level primitives just makes software development easier and the results more reliable.
Lastly, one advise for further questions: As initially mentioned, not all terms have a universal meaning associated with them. Therefore, it would help if you provided additional info, like in particular the programming language you intend to use.
Because you need to be careful when using synchronization. If you abuse of it, you may have performance issues. Using thread-safe collections when possible is usually better for performance and you make sure that you haven't errors or deadlocks.
Many projects I work on have poor threading implementations and I am the sucker who has to track these down. Is there an accepted best way to handle threading. My code is always waiting for an event that never fires.
I'm kinda thinking like a design pattern or something.
(Assuming .NET; similar things would apply for other platforms.)
Well, there are lots of things to consider. I'd advise:
Immutability is great for multi-threading. Functional programming works well concurrently partly due to the emphasis on immutability.
Use locks when you access mutable shared data, both for reads and writes.
Don't try to go lock-free unless you really have to. Locks are expensive, but rarely the bottleneck.
Monitor.Wait should almost always be part of a condition loop, waiting for a condition to become true and waiting again if it's not.
Try to avoid holding locks for longer than you need to.
If you ever need to acquire two locks at once, document the ordering thoroughly and make sure you always use the same order.
Document the thread-safety of your types. Most types don't need to be thread-safe, they just need to not be thread hostile (i.e. "you can use them from multiple threads, but it's your responsibility to take out locks if you want to share them)
Don't access the UI (except in documented thread-safe ways) from a non-UI thread. In Windows Forms, use Control.Invoke/BeginInvoke
That's off the top of my head - I probably think of more if this is useful to you, but I'll stop there in case it's not.
Learning to write multi-threaded programs correctly is extremely difficult and time consuming.
So the first step is: replace the implementation with one that doesn't use multiple threads at all.
Then carefully put threading back in if, and only if, you discover a genuine need for it, when you've figured out some very simple safe ways to do so. A non-threaded implementation that works reliably is far better than a broken threaded implementation.
When you're ready to start, favour designs that use thread-safe queues to transfer work items between threads and take care to ensure that those work items are accessed only by one thread at a time.
Try to avoid just spraying lock blocks around your code in the hope that it will become thread-safe. It doesn't work. Eventually, two code paths will acquire the same locks in a different order, and everything will grind to a halt (once every two weeks, on a customer's server). This is especially likely if you combine threads with firing events, and you hold the lock while you fire the event - the handler may take out another lock, and now you have a pair of locks held in a particular order. What if they're taken out in the opposite order in some other situation?
In short, this is such a big and difficult subject that I think it is potentially misleading to give a few pointers in a short answer and say "Off you go!" - I'm sure that's not the intention of the many learned people giving answers here, but that is the impression many get from summarised advice.
Instead, buy this book.
Here is a very nicely worded summary from this site:
Multithreading also comes with
disadvantages. The biggest is that it
can lead to vastly more complex
programs. Having multiple threads does
not in itself create complexity; it's
the interaction between the threads
that creates complexity. This applies
whether or not the interaction is
intentional, and can result long
development cycles, as well as an
ongoing susceptibility to intermittent
and non-reproducable bugs. For this
reason, it pays to keep such
interaction in a multi-threaded design
simple – or not use multithreading at
all – unless you have a peculiar
penchant for re-writing and debugging!
Perfect summary from Stroustrup:
The traditional way of dealing with concurrency by letting a bunch of
threads loose in a single address space and then using locks to try to
cope with the resulting data races and coordination problems is
probably the worst possible in terms of correctness and
comprehensibility.
(Like Jon Skeet, much of this assumes .NET)
At the risk of seeming argumentative, comments like these just bother me:
Learning to write multi-threaded
programs correctly is extremely
difficult and time consuming.
Threads should be avoided when
possible...
It is practically impossible to write software that does anything significant without leveraging threads in some capacity. If you are on Windows, open your Task Manager, enable the Thread Count column, and you can probably count on one hand the number of processes that are using a single thread. Yes, one should not simply use threads for the sake of using threads nor should it be done cavalierly, but frankly, I believe these cliches are used too often.
If I had to boil multithreaded programming down for the true novice, I would say this:
Before jumping into it, first understand that the the class boundary is not the same as a thread boundary. For example, if a callback method on your class is called by another thread (e.g., the AsyncCallback delegate to the TcpListener.BeginAcceptTcpClient() method), understand that the callback executes on that other thread. So even though the callback occurs on the same object, you still have to synchronize access to the members of the object within the callback method. Threads and classes are orthogonal; it is important to understand this point.
Identify what data needs to be shared between threads. Once you have defined the shared data, try to consolidate it into a single class if possible.
Limit the places where the shared data can be written and read. If you can get this down to one place for writing and one place for reading, you will be doing yourself a tremendous favor. This is not always possible, but it is a nice goal to shoot for.
Obviously make sure you synchronize access to the shared data using the Monitor class or the lock keyword.
If possible, use a single object to synchronize your shared data regardless of how many different shared fields there are. This will simplify things. However, it may also overly constrain things too, in which case, you may need a synchronization object for each shared field. And at this point, using immutable classes becomes very handy.
If you have one thread that needs to signal another thread(s), I would strongly recommend using the ManualResetEvent class to do this instead of using events/delegates.
To sum up, I would say that threading is not difficult, but it can be tedious. Still, a properly threaded application will be more responsive, and your users will be most appreciative.
EDIT:
There is nothing "extremely difficult" about ThreadPool.QueueUserWorkItem(), asynchronous delegates, the various BeginXXX/EndXXX method pairs, etc. in C#. If anything, these techniques make it much easier to accomplish various tasks in a threaded fashion. If you have a GUI application that does any heavy database, socket, or I/O interaction, it is practically impossible to make the front-end responsive to the user without leveraging threads behind the scenes. The techniques I mentioned above make this possible and are a breeze to use. It is important to understand the pitfalls, to be sure. I simply believe we do programmers, especially younger ones, a disservice when we talk about how "extremely difficult" multithreaded programming is or how threads "should be avoided." Comments like these oversimplify the problem and exaggerate the myth when the truth is that threading has never been easier. There are legitimate reasons to use threads, and cliches like this just seem counterproductive to me.
You may be interested in something like CSP, or one of the other theoretical algebras for dealing with concurrency. There are CSP libraries for most languages, but if the language wasn't designed for it, it requires a bit of discipline to use correctly. But ultimately, every kind of concurrency/threading boils down to some fairly simple basics: Avoid shared mutable data, and understand exactly when and why each thread may have to block while waiting for another thread. (In CSP, shared data simply doesn't exist. Each thread (or process in CSP terminology) is only allowed to communicate with others through blocking message-passing channels. Since there is no shared data, race conditions go away. Since message passing is blocking, it becomes easy to reason about synchronization, and literally prove that no deadlocks can occur.)
Another good practice, which is easier to retrofit into existing code is to assign a priority or level to every lock in your system, and make sure that the following rules are followed consistently:
While holding a lock at level N, you
may only acquire new locks of lower levels
Multiple locks at the same level must
be acquired at the same time, as a
single operation, which always tries
to acquire all the requested locks in
the same global order (Note that any
consistent order will do, but any
thread that tries to acquire one or
more locks at level N, must do
acquire them in the same order as any
other thread would do anywhere else
in the code.)
Following these rules mean that it is simply impossible for a deadlock to occur. Then you just have to worry about mutable shared data.
BIG emphasis on the first point that Jon posted. The more immutable state that you have (ie: globals that are const, etc...), the easier your life is going to be (ie: the fewer locks you'll have to deal with, the less reasoning you'll have to do about interleaving order, etc...)
Also, often times if you have small objects to which you need multiple threads to have access, you're sometimes better off copying it between threads rather than having a shared, mutable global that you have to hold a lock to read/mutate. It's a tradeoff between your sanity and memory efficiency.
Looking for a design pattern when dealing with threads is the really best approach to start with. It's too bad that many people don't try it, instead attempting to implement less or more complex multithreaded constructs on their own.
I would probably agree with all opinions posted so far. In addition, I'd recommend to use some existing more coarse-grained frameworks, providing building blocks rather than simple facilities like locks, or wait/notify operations. For Java, it would be simply the built-in java.util.concurrent package, which gives you ready-to-use classes you can easily combine to achieve a multithreaded app. The big advantage of this is that you avoid writing low-level operations, which results in hard-to-read and error-prone code, in favor of a much clearer solution.
From my experience, it seems that most concurrency problems can be solved in Java by using this package. But, of course, you always should be careful with multithreading, it's challenging anyway.
Adding to the points that other folks have already made here:
Some developers seem to think that "almost enough" locking is good enough. It's been my experience that the opposite can be true -- "almost enough" locking can be worse than enough locking.
Imagine thread A locking resource R, using it, and then unlocking it. A then uses resource R' without a lock.
Meanwhile, thread B tries to access R while A has it locked. Thread B is blocked until thread A unlocks R. Then the CPU context switches to thread B, which accesses R, and then updates R' during its time slice. That update renders R' inconsistent with R, causing a failure when A tries to access it.
Test on as many different hardware and OS architectures as possible. Different CPU types, different numbers of cores and chips, Windows/Linux/Unix, etc.
The first developer who worked with multi-threaded programs was a guy named Murphy.
Well, everyone thus far has been Windows / .NET centric, so I'll chime in with some Linux / C.
Avoid futexes at all costs(PDF), unless you really, really need to recover some of the time spent with mutex locks. I am currently pulling my hair out with Linux futexes.
I don't yet have the nerve to go with practical lock free solutions, but I'm rapidly approaching that point out of pure frustration. If I could find a good, well documented and portable implementation of the above that I could really study and grasp, I'd probably ditch threads completely.
I have come across so much code lately that uses threads which really should not, its obvious that someone just wanted to profess their undying love of POSIX threads when a single (yes, just one) fork would have done the job.
I wish that I could give you some code that 'just works', 'all the time'. I could, but it would be so silly to serve as a demonstration (servers and such that start threads for each connection). In more complex event driven applications, I have yet (after some years) to write anything that doesn't suffer from mysterious concurrency issues that are nearly impossible to reproduce. So I'm the first to admit, in that kind of application, threads are just a little too much rope for me. They are so tempting and I always end up hanging myself.
I'd like to follow up with Jon Skeet's advice with a couple more tips:
If you are writing a "server", and are likely to have a high amount of insert parallelism, don't use Microsoft's SQL Compact. Its lock manager is stupid. If you do use SQL Compact, DON'T use serializable transactions (which happens to be the default for the TransactionScope class). Things will fall apart on you rapidly. SQL Compact doesn't support temporary tables, and when you try to simulate them inside of serialized transactions it does rediculsouly stupid things like take x-locks on the index pages of the _sysobjects table. Also it get's really eager about lock promotion, even if you don't use temp tables. If you need serial access to multiple tables , your best bet is to use repeatable read transactions(to give atomicity and integrity) and then implement you own hierarchal lock manager based on domain-objects (accounts, customers, transactions, etc), rather than using the database's page-row-table based scheme.
When you do this, however, you need to be careful (like John Skeet said) to create a well defined lock hierarchy.
If you do create your own lock manager, use <ThreadStatic> fields to store information about the locks you take, and then add asserts every where inside the lock manager that enforce your lock hierarchy rules. This will help to root out potential issues up front.
In any code that runs in a UI thread, add asserts on !InvokeRequired (for winforms), or Dispatcher.CheckAccess() (for WPF). You should similarly add the inverse assert to code that runs in background threads. That way, people looking at a method will know, just by looking at it, what it's threading requirements are. The asserts will also help to catch bugs.
Assert like crazy, even in retail builds. (that means throwing, but you can make your throws look like asserts). A crash dump with an exception that says "you violated threading rules by doing this", along with stack traces, is much easier to debug then a report from a customer on the other side of the world that says "every now and then the app just freezes on me, or it spits out gobbly gook".
It's the mutable state, stupid
That is a direct quote from Java Concurrency in Practice by Brian Goetz. Even though the book is Java-centric, the "Summary of Part I" gives some other helpful hints that will apply in many threaded programming contexts. Here are a few more from that same summary:
Immutable objects are automatically thread-safe.
Guard each mutable variable with a lock.
A program that accesses a mutable variable from multiple threads without
synchronization is a broken program.
I would recommend getting a copy of the book for an in-depth treatment of this difficult topic.
(source: umd.edu)
Instead of locking on containers, you should use ReaderWriterLockSlim. This gives you database like locking - an infinite number of readers, one writer, and the possibility of upgrading.
As for design patterns, pub/sub is pretty well established, and very easy to write in .NET (using the readerwriterlockslim). In our code, we have a MessageDispatcher object that everyone gets. You subscribe to it, or you send a message out in a completely asynchronous manner. All you have to lock on is the registered functions and any resources that they work on. It makes multithreading much easier.
When I was learning Java coming from a background of some 20 years of procedural programming with basic, Pascal, COBOL and C, I thought at the time that the hardest thing about it was wrapping my head around the OOP jargon and concepts. Now with about 8 years of solid Java under my belt, I have come to the conclusion that the single hardest thing about programming in Java and similar languages like C# is the multithreaded/concurrent aspects.
Coding reliable and scalable multi-threaded applications is just plain hard! And with the trend for processors to grow "wider" rather than faster, it is rapidly becoming just plain critical.
The hardest area is, of course, controlling interactions between threads and the resulting bugs: deadlocks, race conditions, stale data and latency.
So my question to you is this: what approach or methodology do you employ for producing safe concurrent code while mitigating the potential for deadlocks, latency, and other problems? I have come up with an approach which is a little unconventional but has worked very well in several large applications, which I will share in a detailed answer to this question.
This not only applies to Java but to threaded programming in general. I find myself avoiding most of the concurrency and latency problems just by following these guidelines:
1/ Let each thread run its own lifetime (i.e., decide when to die). It can be prompted from outside (say a flag variable) but it in entirely responsible.
2/ Have all threads allocate and free their resources in the same order - this guarantees that deadlock will not happen.
3/ Lock resources for the shortest time possible.
4/ Pass responsibility for data with the data itself - once you notify a thread that the data is its to process, leave it alone until the responsibility is given back to you.
There are a number of techniques which are coming into the public consciousness just now (as in: the last few years). A big one would be actors. This is something that Erlang first brought to the grid iron but which has been carried forward by newer languages like Scala (actors on the JVM). While it is true that actors don't solve every problem, they do make it much easier to reason about your code and identify trouble spots. They also make it much simpler to design parallel algorithms because of the way they force you to use continuation passing over shared mutable state.
Fork/Join is something you should look at, especially if you're on the JVM. Doug Lea wrote the seminal paper on the topic, but many researchers have discussed it over the years. As I understand it, Doug Lea's reference framework is scheduled for inclusion into Java 7.
On a slightly less-invasive level, often the only steps necessary to simplify a multi-threaded application are just to reduce the complexity of the locking. Fine-grained locking (in the Java 5 style) is great for throughput, but very very difficult to get right. One alternative approach to locking which is gaining some traction through Clojure would be software-transactional memory (STM). This is essentially the opposite of conventional locking in that it is optimistic rather than pessimistic. You start out by assuming that you won't have any collisions, and then allow the framework to fix the problems if and when they occur. Databases often work this way. It's great for throughput on systems with low collision rates, but the big win is in the logical componentization of your algorithms. Rather than arbitrarily associating a lock (or a series of locks) with some data, you just wrap the dangerous code in a transaction and let the framework figure out the rest. You can even get a fair bit of compile-time checking out of decent STM implementations like GHC's STM monad or my experimental Scala STM.
There are a lot of new options for building concurrent applications, which one you pick depends greatly on your expertise, your language and what sort of problem you're trying to model. As a general rule, I think actors coupled with persistent, immutable data structures are a solid bet, but as I said, STM is a little less invasive and can sometimes yield more immediate improvements.
Avoid sharing data between threads where possible (copy everything).
Never have locks on method calls to external objects, where possible.
Keep locks for the shortest amount of time possible.
There is no One True Answer for thread safety in Java. However, there is at least one really great book: Java Concurrency in Practice. I refer to it regularly (especially the online Safari version when I'm on travel).
I strongly recommend that you peruse this book in depth. You may find that the costs and benefits of your unconventional approach are examined in depth.
I typically follow an Erlang style approach. I use the Active Object Pattern.
It works as follows.
Divide your application into very coarse grained units. In one of my current applications (400.000 LOC) I have appr. 8 of these coarse grained units. These units share no data at all. Every unit keeps its own local data. Every unit runs on its own thread (= Active Object Pattern) and hence is single threaded. You don't need any locks within the units. When the units need to send messages to other units they do it by posting a message to a queue of the other units. The other unit picks the message from the queue and reacts on that message. This might trigger other messages to other units.
Consequently the only locks in this type of application are around the queues (one queue and lock per unit). This architecture is deadlock free by definition!
This architecture scales extremely well and is very easy to implement and extend as soon as you understood the basic principle. It like to think of it as a SOA within an application.
By dividing your app into the units remember. The optimum number of long running threads per CPU core is 1.
I recommend flow-based programming, aka dataflow programming. It uses OOP and threads, I feel it like a natural step forward, like OOP was to procedural. Have to say, dataflow programming can't be used for everything, it is not generic.
Wikipedia has good articeles on the topic:
http://en.wikipedia.org/wiki/Dataflow_programming
http://en.wikipedia.org/wiki/Flow-based_programming
Also, it has several advantages, as the incredible flexibile configuration, layering; the programmer (Component programmer) has not to program the business logic, it's done in another stage (putting the processing network together).
Did you know, make is a dataflow system? See make -j, especially if you have multi-core processor.
Writing all the code in a multi-threaded application very... carefully! I don't know any better answer than that. (This involves stuff like jonnii mentioned).
I've heard people argue (and agree with them) that the traditional threading model really won't work going into the future, so we're going to have to develop a different set of paradigms / languages to really use these newfangled multi-cores effectively. Languages like Haskell, whose programs are easily parallelizable since any function that has side effects must be explicitly marked that way, and Erlang, which I unfortunately don't know that much about.
I suggest the actor model.
The actor model is what you are using and it is by far the simplest (and efficient way) for multithreading stuff. Basically each thread has a (synchronized) queue (it can be OS dependent or not) and other threads generate messages and put them in the queue of the thread that will handle the message.
Basic example:
thread1_proc() {
msg = get_queue1_msg(); // block until message is put to queue1
threat1_msg(msg);
}
thread2_proc() {
msg = create_msg_for_thread1();
send_to_queue1(msg);
}
It is a tipical example of producer consumer problem.
It is clearly a difficult problem. Apart from the obvious need for carefulness, I believe that the very first step is to define precisely what threads you need and why.
Design threads as you would design classes : making sure you know what makes them consistent : their contents and their interactions with other threads.
I recall being somewhat shocked to discover that Java's synchronizedList class wasn't fully thread-safe, but only conditionally thread-safe. I could still get burned if I didn't wrap my accesses (iterators, setters, etc.) in a synchronized block. This means that I might've assured my team and my management that my code was thread safe, but I might've been wrong. Another way I can assure thread safety is for a tool to analyse the code and have it pass. STP, Actor model, Erlang, etc are some ways of getting the latter form of assurance. Being able to assure properties of a program reliably is/will be a huge step forward in programming.
Looks like your IOC is somewhat FBP-like :-) It would be fantastic if the JavaFBP code could get a thorough vetting from someone like yourself versed in the art of writing thread-safe code... It's on SVN in SourceForge.
Some experts feel the answer to your question is to avoid threads altogether, because it's almost impossible to avoid unforseen problems. To quote The Problem with Threads:
We developed a process that included
a code maturity rating system (with four levels, red, yellow, green, and blue), design reviews, code
reviews, nightly builds, regression tests, and automated code coverage metrics. The portion
of the kernel that ensured a consistent view of the program structure was written in early 2000,
design reviewed to yellow, and code reviewed to green. The reviewers included concurrency experts,
not just inexperienced graduate students (Christopher Hylands (now Brooks), Bart Kienhuis, John
Reekie, and [Ed Lee] were all reviewers). We wrote regression tests that achieved 100 percent code
coverage...
The... system itself began to be widely used, and every use of the system exercised this
code. No problems were observed until the code deadlocked on April 26, 2004, four years later.
The safest approach to design new applications with multi threading is to adhere to the rule:
No design below the design.
What does that mean?
Imagine you identified major building blocks of your application. Let it be the GUI, some computations engines. Typically, once you have a large enough team size, some people in the team will ask for "libraries" to "share code" between those major building blocks. While it was relatively easy in the start to define the threading and collaboration rules for the major building blocks, all that effort is now in danger as the "code reuse libraries" will be badly designed, designed when needed and littered with locks and mutexes which "feel right".
Those ad-hoc libraries are the design below your design and the major risk for your threading architecture.
What to do about it?
Tell them that you rather have code duplication than shared code across thread boundaries.
If you think, the project will really benefit from some libraries, establish the rule that they must be state-free and reentrant.
Your design is evolving and some of that "common code" could be "moved up" in the design to become a new major building block of your application.
Stay away from the cool-library-on-the-web-mania. Some third party libraries can really save you a lot of time. But there is also a tendency that anyone has their "favorites", which are hardly essential. And with each third party library you add, your risk of running into threading problems increases.
Last not least, consider to have some message based interaction between your major building blocks; see the often mentioned actor model, for example.
The core concerns as I saw them were (a) avoiding deadlocks and (b) exchanging data between threads. A lessor concern (but only slightly lessor) was avoiding bottlenecks. I had already encountered several problems with disparate out of sequence locking causing deadlocks - it's very well to say "always acquire locks in the same order", but in a medium to large system it is practically speaking often impossible to ensure this.
Caveat: When I came up with this solution I had to target Java 1.1 (so the concurrency package was not yet a twinkle in Doug Lea's eye) - the tools at hand were entirely synchronized and wait/notify. I drew on experience writing a complex multi-process communications system using the real-time message based system QNX.
Based on my experience with QNX which had the deadlock concern, but avoided data-concurrency by coping messages from one process's memory space to anothers, I came up with a message-based approach for objects - which I called IOC, for inter-object coordination. At the inception I envisaged I might create all my objects like this, but in hindsight it turns out that they are only necessary at the major control points in a large application - the "interstate interchanges", if you will, not appropriate for every single "intersection" in the road system. That turns out to be a major benefit because they are quite un-POJO.
I envisaged a system where objects would not conceptually invoke synchronized methods, but instead would "send messages". Messages could be send/reply, where the sender waits while the message is processed and returns with the reply, or asynchronous where the message is dropped on a queue and dequeued and processed at a later stage. Note that this is a conceptual distinction - the messaging was implemented using synchronized method calls.
The core objects for the messaging system are an IsolatedObject, an IocBinding and an IocTarget.
The IsolatedObject is so called because it has no public methods; it is this that is extended in order to receive and process messages. Using reflection it is further enforced that child object has no public methods, nor any package or protected methods except those inherited from IsolatedObject nearly all of which are final; it looks very strange at first because when you subclass IsolatedObject, you create an object with 1 protected method:
Object processIocMessage(Object msgsdr, int msgidn, Object msgdta)
and all the rest of the methods are private methods to handle specific messages.
The IocTarget is a means of abstracting visibility of an IsolatedObject and is very useful for giving another object a self-reference for sending signals back to you, without exposing your actual object reference.
And the IocBinding simply binds a sender object to a message receiver so that validation checks are not incurred for every message sent, and is created using an IocTarget.
All interaction with the isolated objects is through "sending" it messages - the receiver's processIocMessage method is synchronized which ensures that only one message is be handled at a time.
Object iocMessage(int mid, Object dta)
void iocSignal (int mid, Object dta)
Having created a situation where all work done by the isolated object is funneled through a single method, I next arranged the objects in a declared hierarchy by means of a "classification" they declare when constructed - simply a string that identifies them as being one of any number of "types of message receiver", which places the object within some predetermined hierarchy. Then I used the message delivery code to ensure that if the sender was itself an IsolatedObject that for synchronous send/reply messages it was one which is lower on the hierarchy. Asynchronous messages (signals) are dispatched to message receivers using separate threads in a thread pool who's entire job deliver signals, therefore signals can be send from any object to any receiver in the system. Signals can can deliver any message data desired, but not reply is possible.
Because messages can only be delivered in an upward direction (and signals are always upward because they are delivered by a separate thread running solely for that purpose) deadlocks are eliminated by design.
Because interactions between threads are accomplished by exchanging messages using Java synchronization, race conditions and issues of stale data are likewise eliminated by design.
Because any given receiver handles only one message at a time, and because it has no other entry points, all considerations of object state are eliminated - effectively, the object is fully synchronized and synchronization cannot accidentally be left off any method; no getters returning stale cached thread data and no setters changing object state while another method is acting on it.
Because only the interactions between major components is funneled through this mechanism, in practice this has scaled very well - those interactions don't happen nearly as often in practice as I theorized.
The entire design becomes one of an orderly collection of subsystems interacting in a tightly controlled manner.
Note this is not used for simpler situations where worker threads using more conventional thread pools will suffice (though I will often inject the worker's results back into the main system by sending an IOC message). Nor is it used for situations where a thread goes off and does something completely independent of the rest of the system such as an HTTP server thread. Lastly, it is not used for situations where there is a resource coordinator that itself does not interact with other objects and where internal synchronization will do the job without risk of deadlock.
EDIT: I should have stated that the messages exchanged should generally be immutable objects; if using mutable objects the act of sending it should be considered a hand over and cause the sender to relinquish all control, and preferably retain no references to the data. Personally, I use a lockable data structure which is locked by the IOC code and therefore becomes immutable on sending (the lock flag is volatile).