opentelemetry-rust: Increasing the channel size of batched OtlpTracePipeline - rust

Question

Is it possible to increase the channel size of OpenTelemetry span batch processor?
I'm currently getting OpenTelemetry trace error occurred. cannot send span to the batch span processor because the channel is full error on a high-traffic scenario.
Here is my configuration code:
let mut exporter_metadata = MetadataMap::new();
exporter_metadata.insert(
"api-key",
"<redacted>".parse()?,
);
let exporter = opentelemetry_otlp::new_exporter()
.tonic()
.with_endpoint("https://otlp.nr-data.net:4317")
.with_tls_config(ClientTlsConfig::default())
.with_metadata(exporter_metadata);
let trace_config = opentelemetry::sdk::trace::config()
.with_resource(Resource::new(vec![
KeyValue::new(
opentelemetry_semantic_conventions::resource::SERVICE_NAME,
"worker",
),
KeyValue::new(
opentelemetry_semantic_conventions::resource::SERVICE_INSTANCE_ID,
"dev-instance",
),
KeyValue::new("kind", "server"),
]))
.with_sampler(Sampler::TraceIdRatioBased(1.0));
let tracer = opentelemetry_otlp::new_pipeline()
.tracing()
.with_exporter(exporter)
.with_trace_config(trace_config)
.install_batch(opentelemetry::runtime::Tokio)?;
let otel_filter = Targets::new().with_target("worker", LevelFilter::INFO);
let otel_layer = tracing_opentelemetry::layer()
.with_tracer(tracer)
.with_filter(otel_filter);
I've looked through the documentation, but there was no configuration mentioning the size of the channel.

Answer 1

After digging through the source code, I've come up with the following solution:
In batch_config.rs, declare new trait and implement it for OtlpTracePipeline:
use anyhow::Result;
use core::convert::Into;
use core::option::Option;
use core::option::Option::{None, Some};
use core::result::Result::Ok;
use opentelemetry::sdk::trace::{BatchConfig, BatchSpanProcessor, Config, TraceRuntime};
use opentelemetry::trace::{TraceError, TracerProvider};
use opentelemetry::{global, sdk};
use opentelemetry_otlp::{Error, OtlpTracePipeline, SpanExporter, SpanExporterBuilder};
pub trait BatchConfigurable {
fn install_batch_manual<B, R>(
self,
trace_config: Option<Config>,
exporter_builder: Option<B>,
batch_config: BatchConfig,
runtime: R,
) -> Result<sdk::trace::Tracer, TraceError>
where
B: Into<SpanExporterBuilder>,
R: TraceRuntime;
}
fn build_batch_with_config_and_exporter<R: TraceRuntime>(
exporter: SpanExporter,
batch_config: BatchConfig,
trace_config: Option<Config>,
runtime: R,
) -> sdk::trace::Tracer {
let batch = BatchSpanProcessor::builder(exporter, runtime)
.with_batch_config(batch_config)
.build();
let mut provider_builder = sdk::trace::TracerProvider::builder().with_span_processor(batch);
if let Some(config) = trace_config {
provider_builder = provider_builder.with_config(config);
}
let provider = provider_builder.build();
let tracer =
provider.versioned_tracer("opentelemetry-otlp", Some(env!("CARGO_PKG_VERSION")), None);
let _ = global::set_tracer_provider(provider);
tracer
}
impl BatchConfigurable for OtlpTracePipeline {
fn install_batch_manual<B, R>(
self,
trace_config: Option<Config>,
exporter_builder: Option<B>,
batch_config: BatchConfig,
runtime: R,
) -> Result<sdk::trace::Tracer, TraceError>
where
B: Into<SpanExporterBuilder>,
R: TraceRuntime,
{
let exporter_builder = exporter_builder.map(|b| b.into());
Ok(build_batch_with_config_and_exporter(
exporter_builder
.ok_or(Error::NoExporterBuilder)?
.build_span_exporter()?,
batch_config,
trace_config,
runtime,
))
}
}
And then use it like this:
let batch_config = BatchConfig::default().with_max_queue_size(1048576);
let tracer = opentelemetry_otlp::new_pipeline()
.tracing()
.install_batch_manual(
Some(trace_config),
Some(exporter),
batch_config,
opentelemetry::runtime::Tokio,
)?;
Roughly speaking, install_batch_manual is equivalent to calling .with_exporter().with_trace_config().install_batch, but with customizable BatchConfig option.