The Observability Wars Are Over. OTel Won.

Three years ago, the observability space was a warzone. Datadog vs New Relic vs Dynatrace. OpenTracing vs OpenCensus vs vendor-specific agents. Every company had a different stack, and switching vendors meant re-instrumenting everything.

Then OpenTelemetry (OTel) hit v1.0 for traces, then metrics, then logs. Vendors fell in line. By 2026, OTel is the default instrumentation standard for any serious distributed system.

The benefits compound:

• Instrument once, export anywhere
• Vendor-neutral data format
• Massive ecosystem of auto-instrumentation
• CNCF-backed, not going anywhere

This post covers what you need to know to use OTel effectively in production today.

Photo by Carlos Muza on Unsplash

The Three Pillars, Unified

OpenTelemetry covers all three observability signals:

The magic is correlation: a trace can link to the logs emitted during that trace, and to the metrics that spiked during that request. One unified data model.

The OTel Architecture

Your Application
     │
     ▼
[OTel SDK] ← auto-instrumentation + manual instrumentation
     │
     ▼
[OTel Collector] ← receive, process, export
     │
     ├──► Jaeger (traces)
     ├──► Prometheus (metrics)
     ├──► Loki (logs)
     ├──► Datadog
     ├──► Honeycomb
     └──► Any OTLP-compatible backend

The OTel Collector is the key piece: a vendor-neutral proxy that receives telemetry from your apps, processes/enriches it, and forwards to any backend. Change backends by updating collector config — zero code changes.

Getting Started: Auto-Instrumentation

The fastest path to value: zero-code auto-instrumentation.

Python (FastAPI example)

pip install opentelemetry-sdk opentelemetry-instrumentation-fastapi opentelemetry-exporter-otlp

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor

# Configure SDK
provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317"))
)
trace.set_tracer_provider(provider)

# One line to instrument FastAPI
FastAPIInstrumentor.instrument_app(app)

Every HTTP request now gets a trace automatically. Incoming/outgoing calls, DB queries, Redis operations — all captured without writing a single span manually.

Node.js

// otel.js - load before your app
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: 'http://otel-collector:4317',
  }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

# Start with auto-instrumentation
node --require ./otel.js app.js

Java (Spring Boot)

# Download the OTel Java agent
curl -L https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar -o otel-agent.jar

# Run with agent - zero code changes needed
java -javaagent:otel-agent.jar \
  -Dotel.exporter.otlp.endpoint=http://otel-collector:4317 \
  -Dotel.service.name=payment-service \
  -jar app.jar

The Java agent instruments 80+ libraries automatically: Spring, Hibernate, gRPC, Kafka, Redis, and more.

Manual Instrumentation: Adding Business Context

Auto-instrumentation captures infrastructure. Manual spans add business context.

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

async def process_payment(order_id: str, amount: float):
    with tracer.start_as_current_span("process_payment") as span:
        # Add business attributes to the span
        span.set_attribute("order.id", order_id)
        span.set_attribute("payment.amount", amount)
        span.set_attribute("payment.currency", "USD")
        
        try:
            result = await charge_card(order_id, amount)
            span.set_attribute("payment.status", "success")
            span.set_attribute("payment.transaction_id", result.transaction_id)
            return result
            
        except PaymentDeclinedException as e:
            span.set_attribute("payment.status", "declined")
            span.set_attribute("error.type", "payment_declined")
            span.record_exception(e)
            raise

Now when you search for a specific order.id in Jaeger/Honeycomb, you see the entire trace with business context. Debug a specific customer’s failed payment in seconds, not hours.

The OTel Collector: Your Observability Control Plane

The Collector is where the real power lives.

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  # Add resource attributes to all telemetry
  resource:
    attributes:
    - key: deployment.environment
      value: production
      action: insert
  
  # Filter out noisy health check traces
  filter:
    traces:
      span:
        - 'attributes["http.route"] == "/health"'
  
  # Probabilistic sampling - keep 10% of low-value traces
  probabilistic_sampler:
    hash_seed: 22
    sampling_percentage: 10
  
  # Batch for efficiency
  batch:
    timeout: 5s
    send_batch_size: 1000

exporters:
  # Export to multiple backends simultaneously
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  
  prometheus:
    endpoint: "0.0.0.0:8889"
  
  otlp/datadog:
    endpoint: https://trace.agent.datadoghq.com
    headers:
      DD-API-KEY: ${DD_API_KEY}

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [resource, filter, probabilistic_sampler, batch]
      exporters: [otlp/jaeger, otlp/datadog]
    metrics:
      receivers: [otlp]
      processors: [resource, batch]
      exporters: [prometheus]

The Collector handles sampling, filtering, enrichment, and fanout — all without touching your application code.

Sampling Strategies in Production

At scale, you can’t store every trace. Sampling is essential.

Head-Based Sampling (Simple)

Decide at trace start whether to sample. Fast, low overhead.

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

# Sample 5% of all traces
sampler = TraceIdRatioBased(0.05)
provider = TracerProvider(sampler=sampler)

Problem: You sample 5% of normal traces and 5% of error traces. You might miss important errors.

Tail-Based Sampling (Smart)

Keep all interesting traces; sample boring ones.

# Collector tail sampling processor
processors:
  tail_sampling:
    decision_wait: 10s
    policies:
    - name: errors-policy
      type: status_code
      status_code: {status_codes: [ERROR]}  # Keep all errors
    
    - name: slow-traces-policy
      type: latency
      latency: {threshold_ms: 1000}  # Keep traces > 1s
    
    - name: sample-rest
      type: probabilistic
      probabilistic: {sampling_percentage: 2}  # Sample 2% of the rest

Tail-based sampling requires the Collector to buffer spans until the trace completes. More complex, but far more valuable — you never lose an error trace.

Connecting Traces to Logs

The killer feature of OTel in 2026: correlated logs.

import logging
from opentelemetry import trace

class OtelLogHandler(logging.Handler):
    def emit(self, record):
        span = trace.get_current_span()
        if span.is_recording():
            ctx = span.get_span_context()
            record.trace_id = format(ctx.trace_id, '032x')
            record.span_id = format(ctx.span_id, '016x')
        
        # Your existing log handling
        super().emit(record)

Now every log line contains the trace ID. In Grafana/Loki, you can click from a trace span directly to the logs from that span, in the same time window.

The debugging flow becomes:

1. Alert fires → find the trace
2. Trace shows which service is slow → click to logs
3. Logs show the exception → fix the bug

Kubernetes: OTel Operator

For Kubernetes deployments, the OTel Operator handles injection automatically:

# Install operator
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

# Annotate namespace for auto-injection
apiVersion: v1
kind: Namespace
metadata:
  name: production
  annotations:
    instrumentation.opentelemetry.io/inject-java: "true"
    instrumentation.opentelemetry.io/inject-python: "true"
    instrumentation.opentelemetry.io/inject-nodejs: "true"

Every pod in the annotated namespace gets auto-instrumented. No Dockerfile changes. No app code changes. The operator injects the OTel SDK via init containers.

The State of OTel Backends in 2026

The open-source Grafana stack (Loki + Tempo + Mimir + Grafana) is the dominant self-hosted choice. For managed observability without vendor lock-in, Honeycomb has the best developer experience.

Conclusion

OpenTelemetry is no longer optional for distributed systems. It’s the foundation of modern observability.

The adoption path is clear: start with auto-instrumentation (zero code changes), add manual spans for business context, deploy the Collector for vendor flexibility, implement tail-based sampling at scale.

The promise of OTel — instrument once, choose your backend freely — is fully delivered in 2026. The only question is how quickly you adopt it.

The alternative is vendor lock-in and re-instrumentation every time you want to evaluate a new observability tool. The industry made its choice. Now it’s yours.

이 글이 도움이 되셨다면 공감 및 광고 클릭을 부탁드립니다 :)