Best way to simulate a distributed system? - multithreading

Question

I am building out a distributed system where I'll have about 30,000 modules that will interact with each other. Each module will have a copy of the same software and communicate with neighbors to perform some tasks. I am wanting to simulate this, but having trouble with the simulation architecture. My current approach was to create a thread for every module so each module can run asynchronously, but spinning up 30,000 threads does not seem like a realistic solution. Any ideas or direction on how to simulate 30,000 distributed modules would be helpful.

Answer 1

My team uses a home-built simulation environment for our distributed systems. We primarily use it for simulating interactions in a unit test framework (very nice for regression tests!), but it can also be used for long-lived simulations.
Here are the main pieces:
A library that simulates the network and the clock. This library allows us to programmatically stop the "clock" or the "network" and step through either. The network also has hooks to block traffic to/from destinations.
Components are event-driven. They are basically either actors with mailboxes or execution queues (like java's ExecutorService). We don't use an actor framework nor fiber-thread framework. In unit tests we prefer these to be single-threaded, but for simulations we use a single thread pool to run the entire program.
We use dependency injection to swap the real network/clock/threading and the simulated network/clock/threading. (We often bundle these together in an Environment interface.)
Here is a toy example of the environment in action using Paxos:
#Test
public void paxosExample() throws Exception {
// create a simulator, then, in the commented section below, log the trace someplace for later perusal
Network network = Network.simple();
// Uncomment this to log the network trace to a file which can be very useful for debugging.
// network.traceToFile( TRACEFILE );
// log.info( "check out the trace file ", "filename", TRACEFILE );
// create the Paxonians
List<Paxonian> paxonians = IntStream.range(0, N)
.mapToObject(i -> {
SimNic nic = network.provisionNic( Paxosian.NIC_NAME_PREFIX );
return new Paxosian(nic, VALUES[i]);
})
.collect(Collectors.toList());
// start the protocol.
for (Paxosian p : paxosians) {
p.start();
}
log.info("here we go");
network.stepRecursive( StepSelector.RANDOM );
Paxonian first = paxonians.get(0);
assertNotNull( first.getDecision() );
for (Paxosian p : paxosians) {
assertEquals(first.getDecision(), p.getDecision());
}
}