Hazelcast tracing via sleuth - hazelcast

Question

I wonder if there is some integration of sleuth in hazelcast. In my application I have hazelcast queue with event listeners configured for addEntity events and problem is that span seems to be broken once this listener triggeres. I know that there is integration of sleuth for ExecutorService, but is there something similar for com.hazelcast.core.ItemListener? Thanks in advance.
UPD: Giving more details.
I have some sample service that uses both spring-cloud-sleth and hazelcast queue
package com.myapp;
import com.hazelcast.core.Hazelcast;
import com.hazelcast.core.HazelcastInstance;
import com.hazelcast.core.IQueue;
import com.hazelcast.core.ItemEvent;
import com.hazelcast.core.ItemListener;
import java.util.concurrent.Executors;
import lombok.extern.slf4j.Slf4j;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cloud.sleuth.DefaultSpanNamer;
import org.springframework.cloud.sleuth.TraceRunnable;
import org.springframework.cloud.sleuth.Tracer;
import org.springframework.scheduling.annotation.Async;
import org.springframework.stereotype.Service;
#Service
#Slf4j
public class SomeService {
private HazelcastInstance hazelcastInstance =
Hazelcast.newHazelcastInstance();
private IQueue<String> queue = hazelcastInstance.getQueue("someQueue");
private Tracer tracing;
#Autowired(required = false)
public void setTracer(Tracer tracer) {
this.tracing = tracer;
}
{
queue.addItemListener(new ItemListener<String>() {
#Override
public void itemAdded(ItemEvent<String> item) {
log.info("This is span");
log.info("This is item " + item);
}
#Override
public void itemRemoved(ItemEvent<String> item) {
}
}, true);
}
#Async
public void processRequestAsync() {
log.info("Processing async");
log.info("This is span");
Executors.newSingleThreadExecutor().execute(
new TraceRunnable(tracing, new DefaultSpanNamer(), () -> log.info("Some Weird stuff")));
queue.add("some stuff");
}
}
and once I call processRequestAsync I receive following output in console:
INFO [-,792a6c3ad3e91280,792a6c3ad3e91280,false] 9996 --- [nio-8080-exec-2] com.myapp.SomeController : Incoming request!
INFO [-,792a6c3ad3e91280,792a6c3ad3e91280,false] 9996 --- [nio-8080-exec-2] com.myapp.SomeController : This is current span [Trace: 792a6c3ad3e91280, Span: 792a6c3ad3e91280, Parent: null, exportable:false]
INFO [-,792a6c3ad3e91280,7d0c06d3e24a7ba1,false] 9996 --- [cTaskExecutor-1] com.myapp.SomeService : Processing async
INFO [-,792a6c3ad3e91280,7d0c06d3e24a7ba1,false] 9996 --- [cTaskExecutor-1] com.myapp.SomeService : This is span
INFO [-,792a6c3ad3e91280,8a2f0a9028f44979,false] 9996 --- [pool-1-thread-1] com.myapp.SomeService : Some Weird stuff
INFO [-,792a6c3ad3e91280,7d0c06d3e24a7ba1,false] 9996 --- [cTaskExecutor-1] c.h.i.p.impl.PartitionStateManager : [10.236.31.22]:5701 [dev] [3.8.3] Initializing cluster partition table arrangement...
INFO [-,,,] 9996 --- [e_1_dev.event-4] com.myapp.SomeService : This is span
INFO [-,,,] 9996 --- [e_1_dev.event-4] com.myapp.SomeService : This is item ItemEvent{event=ADDED, item=some stuff, member=Member [10.236.31.22]:5701 - b830dbf0-0977-42a3-a15d-800872221c84 this}
So looks like span was broked once we go to eventListener code and I wonder how can I propagate or create new span inside hazelcast queue

Answer 1

Sleuth (at the time of writing) does not support Hazelcast.
The solution is more general than just Hazelcast - you need to pass Zipkin's brave.Span between the client and server, but brave.Span is not serializable.
Zipkin provides a means by which to work around this.
Given a brave.Span on the client, you can convert it to a java.util.Map:
Span span = ...
Map<String, String> map = new HashMap<>();
tracing.propagation().injector(Map<String, String>::put).inject(span.context(), map);
On the server you can convert the java.util.Map back to a brave.Span:
Span span = tracer.toSpan(tracing.propagation().extractor(Map<String, String>::get).extract(map).context())
The use of java.util.Map can obviously be replaced as need be, but the principle is the same.

Answer 2

I can't get it to work for ItemListeners. I think we'd need to be able to wrap Hazelcast's StripedExecutor in something like a LazyTraceThreadPoolTaskExecutor (but one that accepts a plain Executor delegate instead of a ThreadPoolTaskExecutor).
For EntryProcessors, I've hacked this together. A factory to create EntryProcessors, passing in the current span from the thread that creates the processor. When the processor runs, it uses that span as the parent span in the executor thread.
#Component
public class SleuthedEntryProcessorFactory {
private final Tracer tracer;
public SleuthedEntryProcessorFactory(Tracer tracer) {
this.tracer = tracer;
}
/**
* Create an entry processor that will continue the Sleuth span of the thread
* that invokes this method.
* Mutate the given value as required. It will then be set on the entry.
*
* #param name name of the span
* #param task task to perform on the map entry
*/
public <K, V, R> SleuthedEntryProcessor<K, V, R> create(String name, Function<V, R> task) {
return new SleuthedEntryProcessor<>(name, tracer.getCurrentSpan(), task);
}
}
/**
* Copies the MDC context (which contains Sleuth's trace ID, etc.) and the current span
* from the thread that constructs this into the thread that runs this.
* #param <K> key type
* #param <V> value type
* #param <R> return type
*/
#SpringAware
public class SleuthedEntryProcessor<K, V, R> extends AbstractEntryProcessor<K, V> {
private final Map<String, String> copyOfContextMap;
private final String name;
private final Span parentSpan;
private final Function<V, R> task;
private transient Tracer tracer;
public SleuthedEntryProcessor(String name, Span parentSpan, Function<V, R> task) {
this(name, parentSpan, task, true);
}
public SleuthedEntryProcessor(
String name, Span parentSpan, Function<V, R> task, boolean applyOnBackup) {
super(applyOnBackup);
this.name = name + "Hz";
this.parentSpan = parentSpan;
this.task = task;
copyOfContextMap = MDC.getCopyOfContextMap();
}
#Override
public final R process(Map.Entry<K, V> entry) {
if (nonNull(copyOfContextMap)) {
MDC.setContextMap(copyOfContextMap);
}
Span span = tracer.createSpan(toLowerHyphen(name), parentSpan);
try {
V value = entry.getValue();
// The task mutates the value.
R result = task.apply(value);
// Set the mutated value back onto the entry.
entry.setValue(value);
return result;
} finally {
MDC.clear();
tracer.close(span);
}
}
#Autowired
public void setTracer(Tracer tracer) {
this.tracer = tracer;
}
}
Then pass the EntryProcessor to your IMap like this:
Function<V, R> process = ...;
SleuthedEntryProcessor<K, V, R> entryProcessor = sleuthedEntryProcessorFactory.create(label, process);
Map<K, R> results = iMap.executeOnEntries(entryProcessor);