Why goodput matters more than throughput for LLM serving

Pod Logs #1 — July 2026

Your GPUs Are 95% Idle, and Buying More Won’t Fix It

Making Efficiency a Platform Capability

PANEL | When Domains Collide: AI, Security & FinOps Inside the Platform Engineering Stack

ROUNDTABLE | OpenTelemetry in Production

Platform Engineering and DevOps

Transforming optimization into an invisible platform capability

Stop Hiding Kubernetes Pod Inefficiencies With Horizontal Scaling

Practical Policy-as-Code with OPA and a Managed Sidecar Container

Platform Engineering Untold Truths: Is It Just an Infrastructure Matter?

An Open-Source First Step to Greenify Your Cloud Environments

A Policy-as-Code Approach to RBAC Authorization

WORKSHOP | The Game of Platforms: A Guided Journey into the World of Digital Platforms

Challenges and Takeaways of Managing AI Workloads on Cloud

Finding Observability and DevEx Tranquility Sailing the Monitoring Data Seas

From DORA to the SPACE and Beyond: Metrics to Drive Platform Engineering Initiatives

When Platform Engineers Meet SREs: The Birth of Observability-as-a-Service Superpowers

Observability-as-a-Service: When Platform Engineers and SREs Join Forces

Platform Engineers Assemble! Create Your FaaS Platform Using WASM

The Kubernetes Compass: Navigating Space and Time for Energy Efficiency

How Platform Engineering Streamlines the End-to-End Delivery of a Self-Hosted Agentic Architecture

ROUNDTABLE | When Platform Engineers Meet SREs: The Birth of Observability-as-a-Service Superpowers

In API We Trust: Circular Design for Scalable Platforms

Another Cluster Bites the Dust... and That's Just Fine!

Another Cluster Bites the Dust… E Va Bene Così!

Can an AI Agent Give a Talk? Let's Find the Speaker in the Room!

SLMs and the Agents Revolution: Rethinking How Software Is Built

Serving LLMs on Kubernetes: A Golden Path for Platform Engineers

PANEL | The Future of Developer Portals

Cloud Native Live: Kubernetes v1.35 Release

Your Cluster is Lying to ArgoCD (And How to Catch It)

The Self-Improving Platform: Closing the Loop Between Telemetry and Tuning

Beyond Deployment: Mapping the Reality of Day 2 Operations in Platform Engineering

Making Observability Work at the Edge

The Next Chapter of Developer Experience: TAG DevEx in Action

Da Zero a Cento: Costruire un’Infrastruttura che si Auto-Ottimizza

Stop Overprovisioning for Application Startup

Beyond the Data Swamp: Building Your Semantic Observability Mesh


[Published by CNCF](https://www.cncf.io/blog/2026/07/20/why-goodput-matters-more-than-throughput-for-llm-serving/)

When we benchmark an LLM serving setup, the number almost everyone reaches for first is throughput: how many requests per second the system can push through. It is easy to measure, easy to compare, and it lines up nicely with the thing finance actually cares about, which is dollars per request. So we chase it, tuning batching, raising concurrency, squeezing the GPU, and watching requests per second climb.

The trouble is that this number, on its own, can keep rising while your service quietly gets worse. I ran into exactly that while tuning vLLM on a single GPU and the gap was wide enough to be worth walking through because the lesson travels far beyond my particular setup.

## Throughput counts plates, goodput counts happy diners

Picture an LLM endpoint as a restaurant kitchen. Throughput is the number of plates that leave the pass every hour. It tells you the kitchen is busy but it says nothing about whether the food arrived hot, on time or at the right table. A kitchen can double its plates per hour and still send a full room of cold and late dinners.

The metric that captures what we actually want is [goodput](https://haoailab.com/blogs/distserve/). The definition is refreshingly plain: it is the number of completed requests per second that also meet your latency targets. For LLM serving those targets are usually time to first token (TTFT), the wait before anything appears on screen, and time per output token (TPOT), the pace at which the rest of the answer streams out. A request that finishes but blows past either target is a plate that left the kitchen cold. It still counts toward throughput but it does not count toward goodput.

## The setup, and how I drove load

Everything here runs on parts you probably already have. I served Qwen2.5-7B with vLLM on a single NVIDIA A10G with 24 GB of memory inside an EKS cluster, drove load with [GuideLLM](https://github.com/vllm-project/guidellm), and watched the system through Prometheus and Grafana with GPU level metrics coming from NVIDIA’s DCGM exporter. The entire setup is on GitHub, Kubernetes manifests and all, so you can reproduce my runs and, more usefully, aim the same benchmark at your own model and real traffic instead of the synthetic load I used.

One honest note before the numbers. A single A10G is not the GPU you would pick for a production-grade LLM service, so the absolute numbers here will look modest next to an H100 cluster. I picked it on purpose to keep these studies cheap enough to repeat often. The point of this post is not the ceiling, it is the shape of the trade-off between throughput and latency, and that shape holds regardless of how big the GPU is.

I focused on changing only three vLLM settings: `gpu_memory_utilization, the batched token budget (max_num_batched_tokens)`, and the cap on `concurrent sequences (max_num_seqs)`. I picked these three on purpose because together they decide the throughput against latency trade-off, they control how much GPU memory is left for the KV cache, how much work goes into each batch step and how many requests run at the same time. They are also the settings an operator can safely change in production without touching the model, the hardware, or the cluster, so holding everything else fixed keeps the results easy to read and easy to reproduce.

How you send the traffic matters just as much as how you tune it, so it is worth being explicit about it. I used GuideLLM in two ways. A rate sweep ramps the request rate up gradually and lets me read latencies while the server still keeps up, so the numbers reflect real serving and not a backlog. A throughput run does the opposite, firing requests as fast as the server will take them to find the ceiling, which pushes it into saturation. I will use both in this post, and the contrast between them is part of the lesson.

## What the studies show

I ran the same setup against three kinds of traffic because the shape of the traffic decides which latency target actually bites. A chatbot sends a fairly short prompt and gets a fairly short answer. A reasoning workload sends a short prompt and gets a very long answer, on the order of 4,000 tokens. An agentic workload sends many short calls one after another to finish a single task.

| **Workload** | **Prompt / Output (tokens)** | **What the SLO hangs on** |
| --- | --- | --- |
| Chatbot | ~485 / 121 | tight TTFT, medium TPOT |
| Reasoning | ~400 / 4000 | TPOT, with TTFT loose |
| Agentic | ~512 / 128 | per-call TTFT that adds up |

The chatbot study is the clearest case, so start there, and start with what I actually asked it to do. The goal was to maximize combined token throughput, `prefill_token_throughput + decode_token_throughput`, under one constraint: average TTFT had to stay at or below 1.5 seconds. I also added a windowing rule, scoring each config from a stretch of eight consecutive data points where prefill throughput had settled down rather than from any single sample. The reason is simple and worth keeping in mind for your own benchmarks too. A live metric like token throughput jitters as individual requests start and finish so one sample can land on a lucky peak or an unlucky dip. Requiring eight steady points in a row is an easy way to score the system’s real running rate instead of a blip.

You can see that jitter, and the windowing rule doing its job in the raw metrics from two configs I will use as our main example. Both ramp up over the run as GuideLLM raises the load, both wobble step to step and both still let you read off a believable plateau once the ramp settles which is exactly the stretch the windowing rule is built to find.

![Image one of the raw metrics from the two configs used as the main example.](https://www.cncf.io/wp-content/uploads/2026/07/image-13.png)

![Image two of the raw metrics from the two configs used as the main example.](https://www.cncf.io/wp-content/uploads/2026/07/image-15.png)

![Image three of the raw metrics from the two configs used as the main example.](https://www.cncf.io/wp-content/uploads/2026/07/image-16.png)

| **Config** | **gpu_memory_utilization** | **max_num_batched_tokens** | **max_num_seqs** | **TTFT avg** | **Prefill tput** | **Decode tput** |
| --- | --- | --- | --- | --- | --- | --- |
| Exp 1 | 0.89 | 4160 | 212 | 1395 ms | 1925 tok/s | 478 tok/s |
| Exp 20 | 0.87 | 2150 | 271 | 1403 ms | 2934 tok/s | 690 tok/s |

Both configs land at almost the same TTFT, right under the 1.5 second ceiling I set. That is not a coincidence, it is the constraint doing exactly what I asked it to do: hold the first-token wait at the edge of what I said was acceptable then squeeze as much combined throughput out of the rest of the system as it can. Exp 20 wins that squeeze by a wide margin, with about **50 percent more combined throughput** than Exp 1.

That extra throughput is not free, and the bill shows up somewhere I did not constrain. Exp 1’s **p95 TPOT sits at about 50 milliseconds**. Exp 20’s sits at about **494 milliseconds**, nearly ten times higher. I told the search to protect the first token. I never told it to protect the pace of the stream that follows, so it spent that budget without asking. If your target user is reading a chat response as it streams in, a half-second gap between tokens is the difference between a smooth reply and one that visibly stutters even though both configs kept their promise on the metric I actually constrained.

The two latencies move in different directions because they come from two stages of the same request. TTFT comes out of the prefill stage, where the model reads your prompt, and bounding it directly is what keeps both configs near the same 1.4 second mark. Every token after that comes from the decode stage, where vLLM advances all the active requests at once, one token to each per step. Pushing more combined throughput means packing more work into each of those steps, and a fuller step takes longer to run, so every request gets its next token a little later. Picture the kitchen sending one course to every table each round: a fuller room makes each round longer and every table waits more between courses. Constrain how fast the kitchen seats you and brings the first course, and the wait gets pushed onto the gap between the courses that follow instead.  
  
Reasoning and agentic ran a different experiment on purpose. I kept the same goal, maximizing combined prefill and decode throughput, but with no TTFT constraint at all, and I drove load with a throughput run instead of a sweep, sending requests as fast as the server would take them to find the ceiling. The chatbot study answers “how much throughput can we get while keeping the first token fast”. These two answer “how much throughput is actually available”, which is a fair question on its own, as long as you read the latency that comes with the answer rather than assuming it does not matter.

It still does not come for free, and it still varies between configs that post similar throughput. In the agentic study, the two best configs I found were nearly tied on requests per second, 6.20 against 6.11, yet their p95 TPOT differed by about 220 milliseconds, 682 against 902. Same throughput, a noticeably different stream once you are inside one of those many chained calls. In the reasoning study, where each answer runs to about 4,000 tokens, the config with the best raw throughput was not the one you would want once you cared about latency: asking the same search to respect a tighter inter-token target pushed the winner to a different configuration entirely. Three different studies, three different ways of asking the question, and the same answer comes back each time: a throughput number with nothing else attached does not tell you which configuration to run.

## Optimize for the thing you actually sell

Three lessons come out of this for anyone running inference in production.

First, a throughput number with no SLO next to it is marketing, not engineering. The honest target is goodput, meaning requests per second that actually met your latency targets. Count anything else and you are counting cold plates.

Second, the config with the best goodput is not the one you would guess. In my reasoning runs, two settings that looked almost the same ended up far apart and one nearby setting failed outright. The space is full of small cliffs like that, so hand tuning tends to stop at the first number that looks good instead of the best one. The reliable way through it is an automated, data-driven search that tries many configs against your real targets rather than a few hand-picked guesses.

Third, none of this stays put. Traffic, prompts, and models all change over time, and the best config moves with them so the search has to run again, not once.

There is a fourth point worth making even though my study did not test it directly. I changed only three serving knobs and the gap was already this wide. The same trade-off lives in the layers I held fixed too, in the GPU settings, the runtime, and the scheduling, and the largest gains tend to come from tuning the whole stack together rather than one layer at a time since the layers interact.

The practical move is small and worth doing early. Set your TTFT and TPOT targets first, then look for the config that gives you the most throughput while still meeting them instead of chasing throughput on its own. You will often ship a smaller requests-per-second number and a clearly better service, the trade your users would make every time.

If you want to see the gap on your own hardware the full setup is on GitHub at [github.com/graz-dev/vllm-benchmark](https://github.com/graz-dev/vllm-benchmark). Clone it, point GuideLLM at your own model and traffic, set your SLOs, and search for the best goodput rather than the best raw throughput. That gap is usually where your real headroom is hiding.


[Published by Akamas](https://akamas.io/resources/gpu-utilization-llm-inference-optimization/)

AI is reshaping every industry, and agents are at the center of that shift. Many companies are racing to put them to work building assistants that handle customer requests around the clock, pipelines that classify and route data without human input or systems that reason across tools and act on their own. The appeal is obvious. What’s easy to miss in all the excitement is that every one of those agents, no matter how capable it looks from the outside, runs on a model, and that model runs on a GPU that is the engine keeping the whole thing moving.

Enterprises are pouring billions into GPUs to run LLMs and AI agents. For the first time in computing history, we are seeing data-center build-outs at a scale never witnessed before, where the GPU portion alone can account for as much as [70% of total investment](https://www.innovationendeavors.com/insights/future-data-centers), according to recent industry estimates.

Then the first real utilization reports come in. The number is hard to ignore: GPU usage as low as 5%. You might be spending millions on accelerators while [95% of that capacity sits idle](https://akamas.io/resources/gpu-utilization-llm-inference-optimization/#:~:text=AI%20is%20reshaping,the%20same%20hardware.) and yet almost every platform and engineering leader we talk to is trying to do the same thing: buy more GPUs to support the GenAI use cases moving into production this year. **The bottleneck, as it turns out, is usually not capacity but configuration.** On a stack as deep as the GenAI one, the gap between a default setup and a properly tuned one can easily reach 2x throughput or more on the same hardware.

![](https://akamas.io/wp-content/uploads/2026/06/AI-factory-ROI-equals-GPU-utilization.png)

This is the central tension we explored in our recent [webinar](https://akamas.io/events/webinar-genai-optimization-kubernetes/) with Daniele Zonca and Roland Huß from Red Hat, who quite literally wrote the book on running generative AI on Kubernetes.

## We’ve Already Seen This Pattern with CPUs

Here’s a number that should worry anyone treating GPUs the way they treated CPUs. Across one of our customers, with several dozen Kubernetes clusters and around 60,000 CPUs, average utilization over a full day was **5%**. At cloud prices, that adds up to roughly $15 million a year spent on resources that are almost entirely idle.

Those are CPUs. Now apply the same operational habits (over-provisioning, ignoring efficiency, assuming the answer to every capacity question is “add more”) to hardware that costs roughly 40x more per unit and is genuinely scarce. In much of Europe you simply can’t procure H100s on demand, no matter how big your budget is. If 5% utilization on cheap, abundant CPUs costs **$15 million a year**, the equivalent waste on GPUs is an order of magnitude worse. It can quietly block the ROI of an entire GenAI initiative.

## Why GPUs Are So Easy to Waste

GPUs are nothing like CPUs, and that difference is exactly why they’re hard to keep busy.

A **CPU is built for latency**: most of its silicon is dedicated to control logic and caches that predict what comes next, so that a handful of threads can execute general-purpose work as fast as possible otherwise a **GPU is built for throughput**: almost the entire chip is devoted to computation, with thousands of cores optimized for the matrix multiplications that LLM serving depends on. Think of a CPU as a fast motorbike and a GPU as a cargo ship.

The catch is feeding that cargo ship. Every computation requires data to move from high-bandwidth memory (HBM) into the cores, and that transfer frequently becomes the real bottleneck, which is the first reason GPU utilization ends up so low.

This gets sharper once you look at how inference actually runs, across two very different phases:

1. **Prefill** processes the entire input prompt in parallel to produce the first token. It is compute-bound and drives your time to first token (TTFT), which is the waiting time your user feels before anything appears on screen.
2. **Decode** generates tokens one at a time, each step reusing the KV cache built from all previous tokens. It is memory-bound and drives your time per output token (TPOT), which measures how fast the response streams out.

Because the same hardware handles two completely different access patterns, optimizing for one does not automatically improve the other. That mismatch is a core source of complexity in the whole stack.

## The Trade-off You’re Probably Losing On

NVIDIA’s Jensen Huang has popularized a chart worth understanding. When you plot total token throughput (your revenue proxy, since AI services are priced per token) against per-user interactivity (latency), you don’t get a single best point. You get a **Pareto front**: a curve of configurations where improving one dimension always costs you something on the other.

![](https://akamas.io/wp-content/uploads/2026/06/Extracting-2x-throughput-from-the-same-GPUs.png)

Push for maximum throughput and single-user latency suffers. Push for fast responses and you sacrifice a large share of GPU capacity.

Your use case fixes one constraint on that curve. A chatbot needs to emit tokens faster than human reading speed (around 200ms per token) or it feels sluggish and users leave. That means you have a fixed latency limit you cannot cross.

But along that line sit hundreds of possible configurations, and most teams are unknowingly sitting on a suboptimal one. A single misconfiguration can cut effective throughput in half, leading teams to conclude they’re maxed out and order more GPUs, when in reality they’re using only half of what they already paid for.

With proper tuning, doubling the throughput from existing hardware is not unusual. That is the real opportunity.

## The Playbook: Getting 2x from the GPUs You Already Have

The GenAI stack has hundreds of tuning knobs, from the model down to the kernel and Kubernetes itself. That depth is the real challenge. The skills needed to navigate it effectively are genuinely hard to find, which means most teams end up working with defaults that no one has had the time or expertise to question. Here are the levers with the highest payoff.

![](https://akamas.io/wp-content/uploads/2026/06/The-full-stack-LLM-inference-optimization-layers.png)

1. **Right-size the model, then quantize it**

Defaulting to the biggest model for every request is rarely the right call. Smaller and reasoning-tuned models can match quality on the right tasks, and a well-designed agent framework can maintain accuracy while running a smaller model. Quantization can compress a model to as little as 25–30% of its original size while preserving most of its accuracy, which is a significant reduction in resource use for almost no quality cost.

2. **Route intelligently and exploit cache locality**

Pick the right model per request, but also send follow-up messages to the replica that already holds the relevant KV cache. Multi-turn chat and agentic flows send the whole growing conversation back and forth, so cache-aware routing is critical to avoid recomputing work you’ve already done.

3. **Tune the inference engine and the parallelism strategy**

The engine layer around vLLM (or any other inference server) is evolving fast. It pays to match the parallelism strategy to your model:

- *Data parallelism* (replicas behind a load balancer) scales concurrency but not single-request latency.
- *Pipeline parallelism* splits a too-big model across nodes by layers.
- *Tensor parallelism* splits within a node using NVLink/NVSwitch to actually improve latency.

You can combine pipeline and tensor parallelism to get the best of both, but neither is free of trade-offs. Pipeline parallelism is constrained by inter-node network latency, which can be 25–50x slower than staying on a single node. Tensor parallelism introduces its own synchronization overhead across the NVLink fabric inside the node. The right combination always needs to be validated against your specific setup rather than assumed to work out of the box.

4. **Match hardware to the bottleneck**

If your constraint is in the decode phase, the real problem is memory bandwidth. Buying the GPU with the most teraflops will not help. Disaggregated serving (separating prefill and decode pools, as projects like llm-d enable) lets you scale each phase independently and route large inputs to dedicated prefill instances. As inference scales out, expect training-grade networking such as RDMA and InfiniBand to start mattering for inference as well.

5. **Partition GPUs with care**

If a GPU is allocated but underused, you may be able to host a second model on it, but GPUs have both a compute side and a bandwidth side. Partitioning can create a noisy-neighbor effect: spare capacity on paper does not guarantee you can add load without degrading both workloads. Always validate the impact before packing more onto a single card.

6. **Measure what you optimize**

None of these changes are safe to make without proper visibility. Track LLM-specific metrics (TTFT, TPOT), lean on the OpenTelemetry semantic conventions emerging for GenAI, and use tracing tools like MLflow to follow non-deterministic, multi-hop agentic requests. This also gives you the data for internal cost allocation across teams.

GPU-level metrics deserve special attention because they are significantly harder to interpret than application-level ones. Tools like NVIDIA’s DCGM give you utilization numbers, but those numbers are often too coarse to identify whether your actual bottleneck is compute, memory bandwidth, or something else. GPU memory bandwidth, for instance, cannot be directly measured from DCGM at all: vLLM can expose an estimation through the –enable-mfu-metrics flag, but it remains an approximation. To make things more confusing, “GPU utilization” means very different things depending on which layer of the stack you’re looking at (allocation, kernel execution, or model FLOP/s), and conflating them leads to wrong conclusions.

7. **Load test with your own traffic, not a vendor benchmark**

Traditional tools like k6, JMeter, and Locust work at the application level, but purpose-built tools like GuideLLM (part of the vLLM project) let you build realistic, record-and-replay workloads that reflect how your system is actually used. The most important point here is to always build your test dataset from real traffic, because a vendor’s benchmark on a foundation model says nothing about how the system will behave under your specific workload. The optimization space for GenAI is too workload-dependent for generic results to transfer.

## Too Many Knobs for a Human to Tune

With hundreds of knobs spread across three fundamentally different layers (the model, the inference server, and the GPU itself), millions of possible combinations, and a target that shifts with every new use case and every new model release, this is not a one-off exercise you do at launch. It is a continuous practice.

Each layer also requires different expertise. The people who understand model selection and quantization are rarely the same people who can tune vLLM internals or reason about GPU memory bandwidth, and finding someone who spans all three is genuinely uncommon. That makes this well beyond what any team can manage by hand.

The emerging answer is to use AI to optimize AI systems: autonomous tuning that explores the configuration space and finds the settings that maximize throughput while staying within your latency and cost targets. This is exactly the problem [Akamas](https://akamas.io/akamas-ai-infrastructure-optimization-launch/) was built to solve, first for traditional Kubernetes workloads and now for GenAI inference, and we have something significant coming on that front that we will be sharing very soon.

## The Bottom Line

As enterprises move AI agents into production and pour significant capital into the GPU infrastructure to support them, the pressure to show returns on that investment is growing fast. But utilization tells a different story, and the 5% figure is not a hardware problem. It is what happens when a genuinely complex stack gets deployed with default settings and no one has the time to tune it properly.

The good news is that the same depth that makes the GenAI stack hard to configure also creates real headroom. The difference between where most teams are parked today and what their existing hardware can actually deliver is large enough to change the ROI equation entirely.

You do not need more GPUs to get there. You need visibility into your actual bottleneck, a realistic picture of your workload, and a systematic way to explore the configuration space. Everything else in this article follows from those three things.







[Published by PlatformEngineering](https://platformengineering.org/blog/transforming-optimization-into-an-invisible-platform-capability)

In the high-stakes world of cloud-native delivery, organizations are frequently paralyzed by an "invisible wall" that fundamentally separates FinOps professionals, Site Reliability Engineers, and Developers. 

This structural friction strongly resembles a dysfunctional michelin-starred kitchen where the night’s success depends on characters who rarely see eye-to-eye on the operational priorities required to serve a flawless meal. 

![](https://cdn.prod.website-files.com/6489e23dd070ba71d41a33b2/6a0c8288c5dd8bfc13916685_Screenshot%202026-05-19%20at%2017.32.11.png)

Because these highly specialized teams speak entirely different languages regarding risk and reward, this dynamic creates a zero-sum game where the organization must continuously trade off cost for reliability or speed for efficiency. 

When a FinOps spreadsheet indicates that a Kubernetes pod is mostly idle, the SRE immediately perceives a threat to their carefully calculated reliability buffer, while the developers simply lack the contextual data to know how much compute space their specific application actually requires under realistic load. 

This operational paralysis is further exacerbated by the shared ownership model prevalent in many modern enterprises, where a lack of singular accountability means that reactive cost-cutting and conservative over-provisioning remain the default strategies.

## The day 2 operational gap

To break this destructive cycle, we must stop treating operational efficiency as a manual "spring cleaning" task and start recognizing it as an essential requirement that must be embedded deeply within modern platform architecture. 

While the platform engineering community has excelled at creating golden paths to streamline the initial developer experience and deployment pipelines, the lifecycle of applications reaching production has been largely neglected. Currently, our primary method of assisting developers with post-deployment optimization relies on providing access to passive monitoring dashboards rather than delivering automated, actionable intelligence. 

Organizations are striving to run highly performant applications while simultaneously maintaining strict cost controls and ensuring absolute reliability, but static golden paths are simply insufficient for managing this delicate trade-off. 

When building deployment pipelines, everyone shares the single goal of releasing software, which allows for strict and opinionated automation. However, production optimization requires a highly configurable approach because different applications inherently require dynamic tuning strategies tailored to their specific risk profiles and performance needs. For these reasons, static deployment pipelines inevitably fall short during Day 2 operations and lead to systemic resource waste.

## The full-stack optimization puzzle

True optimization represents a complex multi-layered puzzle that extends far beyond simply resizing Kubernetes pods or shutting down idle development environments after business hours. 

Deep efficiency must begin at the application runtime level by tuning elements like the Java Virtual Machine heap size or garbage collection settings to ensure the core workload operates at peak health. Once the runtime is properly configured and understood, engineers can accurately size the container requests and limits to perfectly encase that specific application without causing artificial throttling or fatal memory errors. 

This layered approach must also carefully incorporate how you scale horizontally and the underlying node groups to ensure the entire system remains stable when infrastructure density increases. Without looking at this vertical stack of dependencies, scaling policies might rely on superficial metrics that do not align with the newly established pod boundaries.

![](https://cdn.prod.website-files.com/6489e23dd070ba71d41a33b2/6a0c82b03624c5b2b527b085_Screenshot%202026-05-19%20at%2017.32.51.png)

This inevitably causes the system to fluctuate unpredictably and degrade the performance of your applications. You cannot effectively optimize a system if you treat the container as an opaque black box, yet industry data shows that very few companies actually attempt to tune the container and the runtime in tandem.

### Resolving the kitchen conflict

To solve this puzzle, we must address the conflicting motivations of the specialized roles mentioned earlier. The "invisible wall" persists because each player in the kitchen has a different definition of success:

- The **FinOps** team acts as the restaurant owner who scrutinizes every cloud expenditure and demands higher utilization rates to justify the infrastructure investments.
- The **Site Reliability Engineers** operate as the executive chefs who prioritize absolute uptime and system stability, often demanding massive resource buffers to absorb unexpected traffic spikes.
- The **platform team** serves as the overwhelmed manager attempting to standardize infrastructure while keeping the peace.
- The **developers** act as line cooks who desperately try to innovate without clear guidance on the resource constraints they must operate within.

## Integrating efficiency into the GitOps workflow

By integrating performance gates directly into the continuous integration and continuous deployment pipeline, the platform team establishes a controlled environment where efficiency becomes a core component of the software release lifecycle. 

These purpose-built platform capabilities can continuously analyze historical observability data to generate intelligent recommendations based on predefined tuning profiles that strictly respect the safety limits established by the SRE team.

![](https://cdn.prod.website-files.com/6489e23dd070ba71d41a33b2/6a0c82cefc9e4e1f6cdac159_Screenshot%202026-05-19%20at%2017.33.24.png)

The most effective way to implement these data-driven recommendations involves establishing a human-in-the-loop workflow driven entirely through established GitOps principles. Whenever the platform identifies a more efficient configuration that satisfies both cost constraints and performance requirements, it seamlessly opens a pull request directly in the developer's repository containing the evidence-based sizing adjustments. 

Developers can easily review these insights within their existing tools and approve the new configuration with absolute confidence, knowing that the proposed changes have been mathematically validated against the organizational reliability standards. 

By embedding these automated right-sizing mechanisms directly into the golden path, the platform truly becomes a unified bridge that eliminates the historical friction between engineering departments. This approach moves the organization beyond simple infrastructure provisioning to create a self-improving ecosystem where cost-awareness and performance are continuously balanced without requiring constant human intervention.

By transforming optimization from a manual chore into an invisible platform capability, we solve the "Michelin kitchen" dilemma. This shift allows developers to focus entirely on innovation and feature delivery while the underlying system ensures the organizational menu is always perfectly optimized for both the business and the end user. When efficiency is automated, the invisible wall disappears, replaced by a seamless path toward sustainable, high-performance delivery.


[Published by CNCF co-authored with Hélia Barroso, Alessandro Pomponio, Sonali Srivastava](https://www.cncf.io/blog/2026/04/29/the-state-of-ai-in-cncf-projects-a-first-look-at-the-data/)

At **CNCF TAG Developer Experience**, we recently set out to understand how Artificial Intelligence is shaping open-source development. The response from the community has been impressive in its scale, with nearly half of our initial responses arriving within the first week alone. This immediate engagement highlights the urgency of the topic and the community’s desire for shared guidelines for AI-assisted development.

The data collected from 133 respondents so far represents **nearly 100 unique projects**, giving us confidence that these findings reflect the cloud-native ecosystem at large rather than a narrow subset of projects. This article serves as a sneak peek into our initial findings; a more comprehensive analysis will follow as we process further data.

### Who is answering?

The feedback primarily reflects the perspectives of those on the front lines. The vast majority of participants are code-centric contributors focusing on submission, CI/CD, and infrastructure, while approximately 20% combine engineering with critical roles like release management and documentation.

### How maintainers use AI: Tools and workflows

Modern AI tools have moved beyond simple web chatbots and are now deeply integrated into daily routines. Nearly half of respondents actively use AI assistants directly within their IDEs or command-line interfaces.

![A blue bar graph showing the levels of basic, advanced, other and intermediate. ](https://www.cncf.io/wp-content/uploads/2026/04/image-14.png)

A blue bar graph showing the levels of basic, advanced, other and intermediate.

When it comes to tooling, Claude Code and Github Copilot emerge as clear leaders in the space. Interestingly, only a small fraction (roughly 10%) of contributors still rely on basic chatbots via manual copy-pasting. Meanwhile, a similar percentage of advanced users has already moved toward “high-level integration,” where AI is built directly into project automation for PR reviews and issue triaging.

### Where AI helps the most

Contributors are seeing the most significant boosts in productivity within a few specific areas:

- Writing and refactoring code.
- Improving documentation and debugging.
- Understanding unfamiliar codebases.
- Analyzing Pull Requests.

The high ranking of “understanding the codebase” suggests that AI is acting as a knowledgeable guide, helping developers navigate the inherent complexity of large-scale projects.

### The gap between AI use and official policies

One of our most striking findings is the disconnect between individual AI usage and formal project governance. While local AI use is widespread, the adoption of official policies has lagged behind.

Roughly two-thirds of respondents are either unaware of any specific AI guidelines or confirmed that no official policies exist in their main repositories. Furthermore, the vast majority of projects make no mention of AI usage in their public-facing documentation or contributing guides. While a few pioneering projects are setting the pace with clear policies, the ecosystem is largely operating in an environment that is still figuring out how to govern automated code generation.

![An orange bar graph showing levels of clear policies, informal acceptance and unaware/no policy ](https://www.cncf.io/wp-content/uploads/2026/04/image-15.png)

An orange bar graph showing levels of clear policies, informal acceptance and unaware/no policy

### Community sentiment and code reviews

Despite the lack of formal rules, the general “vibe” toward AI is open and accepting. Roughly one-third of contributors noted that AI usage is generally allowed. Conversely, only a tiny minority (less than 4%) reported that AI usage is explicitly prohibited in their environments.

![A green bar graph showing levels of 'unsure', 'not required', 'specific cases only' and 'always require disclosure.' ](https://www.cncf.io/wp-content/uploads/2026/04/image-16.png)

A green bar graph showing levels of 'unsure', 'not required', 'specific cases only' and 'always require disclosure.'

This pragmatic approach extends to how maintainers handle AI-generated contributions:

- A solid majority follow their standard review process without applying special filters.
- Over a quarter of maintainers prefer a collaborative approach, asking contributors to refine AI-generated code to meet quality standards rather than rejecting it.
- Only a nominal percentage automatically reject suspected AI PRs.

### Top concerns and the call for transparency

While the community is optimistic, several valid concerns remain. Maintainers are particularly worried about:

1. Security vulnerabilities
2. License compliance.
3. The burden on reviewers, caused by **a potential flood of low-effort PRs**.

To mitigate these risks, there is a strong collective desire for visibility. Over half of respondents believe that AI-assisted contributions should always require formal disclosure (such as an “AI-authored” tag). An additional 20% feel disclosure should be required in specific cases. This suggests that while maintainers are willing to accept AI-generated code, they want the transparency necessary to adjust their review efforts accordingly.

### Wrapping up

This first batch of data confirms that AI integration is no longer a trend, it is a core part of the modern workflow. As we move forward, the challenge for the cloud-native community will be balancing this new productivity with the high standards of security and manual oversight that enterprise-grade open source requires.

We aren’t finished yet! To ensure our final report truly reflects the diverse cloud-native landscape, we need your voice. The survey will remain open until Monday, May 18 (End of Day, Anywhere on Earth).If you haven’t shared your experience yet, we invite you to [**contribute to the survey**](https://forms.gle/tBjQp9sFnBF3CXE17) and help us build a more accurate and comprehensive picture of AI’s role in our community.


[Published by Akamas](https://akamas.io/resources/bridging-finops-sre-platform-teams-kubernetes/)

In the world of fine dining, the tension in the kitchen is palpable long before the first guest arrives. Imagine a prestigious Michelin-starred restaurant where the night’s success depends on **four formidable characters who rarely see eye-to-eye**. There is the Owner, who meticulously tracks the cost of every truffle and the electricity bill of the walk-in freezer; there is the Executive Chef, who would rather throw away ten gallons of perfectly good consommé than risk serving a single plate that is less than transcendent; there is the restaurant manager, who struggles to coordinate the chaotic flow between the ovens and tables using a reservation system that feels increasingly outdated; and finally, there are the Line Cooks, the ones actually preparing the dishes.  
  
If the Owner cuts the budget too thin, the quality of the food collapses. If the Chef over-prepares to ensure they never run out of ingredients, the restaurant bleeds money through food waste. If the Manager fails to sync the two, the entire experience becomes a slow-motion disaster. The Line Cooks are caught in the middle: they want to innovate and serve great food, but they lack the data to know if they are using too many expensive ingredients or if their “stove” is set to the right temperature.

In the digital landscape of cloud-native enterprises, our Kubernetes clusters are the kitchens, and our microservices are the delicate dishes being served to millions of hungry users. The **FinOps** team acts as the Owner, scrutinizing the cloud bill and demanding that we stop paying for “tables” that sit empty during off-peak hours. The **Site Reliability Engineers (SREs)** are our Chefs, their reputation is built on uptime and performance, and for them, an over-provisioned cluster is not waste: it is the extra pantry stock that prevents a catastrophic “sold out” sign during a traffic surge. Meanwhile, the Platform Team is the Manager, trying to build a standardized infrastructure that keeps the peace while the tickets pile up. In this scenario, the Developers are the Line Cooks creating the recipes: without clear guidance, they end up requesting more fridge space than necessary, fearing that the Chef (SRE) or the Owner (FinOps) will take away the “heat” they need to finish the service.

## The Invisible Wall

This structural gap is what we call the ***Invisible Wall***. It is a byproduct of specialization where everyone is doing their job perfectly, yet the organization as a whole is failing to achieve true efficiency. According to the [2026 FinOps Foundation’s State of FinOps](https://data.finops.org/#priorities) report, the primary challenge for organizations has shifted toward reducing waste and over-provisioning while still struggling with “empowering engineers to take action”.  
  
This isn’t because engineers are lazy; it is because the Merchant, the Chef and the Line Cooks speak different languages. When a FinOps spreadsheet says a pod is 80% idle, the SRE hears a threat to their reliability buffer while Developers simply lack the data to know how much space their specific application actually requires. Without a shared source of truth that translates “dollars saved” into “risk managed”, these teams remain locked in a cycle of conservative over-provisioning and reactive cost-cutting.

![Diagram showing how FinOps, SRE, Developers and Platform teams' conflicting priorities lead to over-provisioning and friction in Kubernetes environments](https://akamas.io/wp-content/uploads/2026/04/invisible-wall-finops-sre-devs-platform-kubernetes.png.png)

Diagram showing how FinOps, SRE, Developers and Platform teams' conflicting priorities lead to over-provisioning and friction in Kubernetes environments

The tragedy of this friction is that it treats optimization as a zero-sum game, a world where you must choose between a healthy bottom line and a reliable service. Ironically, this defensive over-provisioning doesn’t just inflate the bill; it often results in suboptimal performance and reliability issues, such as unexpected Out-Of-Memory (OOM) errors, as teams lack the precision to balance resources correctly. However, the reality of modern cloud-native architecture is far more nuanced. True efficiency is not about buying cheaper ingredients: it is about the fit of the resource to the task. That is the very principle on which [Akamas Insights](https://akamas.io/insights/) was born. To achieve adequate **fit** for purpose, [Akamas Insights](https://akamas.io/insights/) introduces the concept of the ***Tuning Profiles***. Think of it as the “House Rules” or the “Recipe Standard” for the kitchen. A Tuning Profile allows the Platform Team and the SREs to define the boundaries of optimization specifying which objective matters most (such as cost reduction or latency improvement) and which constraints must be respected (such as minimum replicas or memory headroom). By establishing these profiles, the Chef (SRE) can trust that the “Owner’s” demands will never violate the fundamental safety limits of the service.

![Diagram showing how Akamas Insights aligns FinOps, SRE, and Platform teams around a shared view to achieve high reliability and low waste in Kubernetes](https://akamas.io/wp-content/uploads/2026/04/akamas-insights-shared-truth-finops-sre-platform.png-2048x685.png)

Diagram showing how Akamas Insights aligns FinOps, SRE, and Platform teams around a shared view to achieve high reliability and low waste in Kubernetes

## The Full-Stack Recipe

Instead of looking at a Kubernetes cluster as a collection of static boxes, Akamas Insights treats the environment as a living, breathing organism. By bridging the gap between infrastructure metrics and application runtime behavior, it allows the Owner, the Chef, the Manager and the Line Cooks to finally look at the same dashboard and see the same truth.

When we talk about deep, **full-stack optimization**, we are looking at the vertical stack of dependencies. Optimization is a multi-layered puzzle that starts at the Application Runtime (tuning JVM heap or Go GC) to ensure the engine is healthy. Only then do we move to the Pods, sizing requests and limits to perfectly encase that runtime. Crucially, this is where we also align the Horizontal Pod Autoscaler (HPA). If an HPA is scaling based on a metric that doesn’t align with the Pod’s new limits, the system will “flap” and become unstable. Akamas Insights coordinates these layers to ensure that when we increase density, the contents still have room to breathe.

Finally, we size the Node Group accordingly, ensuring the underlying hardware is the right vessel for these optimized and correctly-scaled containers**.** By following this sequence, the infrastructure becomes a direct reflection of the application’s actual needs, rather than a collection of oversized safety nets.

![Three-layer model showing how full-stack Kubernetes optimization spans App Runtime (JVM, concurrency), Pods and HPA (CPU, memory, scaling), and Node Groups to achieve optimal density](https://akamas.io/wp-content/uploads/2026/04/full-stack-kubernetes-optimization-layers-jvm-pods-nodes.png-2048x545.png)

Three-layer model showing how full-stack Kubernetes optimization spans App Runtime (JVM, concurrency), Pods and HPA (CPU, memory, scaling), and Node Groups to achieve optimal density

## Action over Analysis

This level of insight transforms the conversation from one of ***cutting*** to one of ***tuning***. In our restaurant metaphor, this is the equivalent of a smart inventory system that knows exactly how many guests are coming, their dietary preferences, and the precise temperature each oven needs to be to minimize energy without losing a degree of heat. When the Platform Team can present an optimization plan **backed by a verified Tuning Profile** that accounts for JVM heap settings, CPU cores, and node affinity, the SRE no longer feels the need to hoard resources “just in case”. They can see, through data-driven and historical analysis, that the new configuration is actually more stable than the old, bloated one.

Furthermore, the industry is beginning to realize that the “Spring Cleaning” approach to cloud costs is fundamentally broken. You cannot achieve a Michelin star by cleaning your kitchen once a year: it requires constant, meticulous attention to detail. Yet, many organizations still treat FinOps as a quarterly audit and a reactive “Crawl” stage behavior. According to the [FinOps Foundation’s Workload Optimization](https://www.finops.org/framework/capabilities/workload-optimization/) capability, true maturity (the “Run” stage) is defined by a shift toward **Continuous Optimization**.

However, the path to this maturity is not about handing the keys of the kitchen over to an unpredictable autonomous agent. In a world where reliability is paramount, a “black-box” real-time approach can introduce more instability and “flapping” than it solves. True excellence requires a **Human-in-the-Loop** model where the Developers (the Line Cooks) are the essential piece. Instead of chasing every micro-fluctuation, the most effective optimization happens in lockstep with the **release cycle**.

![Workflow diagram showing the continuous optimization cycle with Akamas Insights: deploy, analyze telemetry, generate AI-powered recommendations, human review, and apply to next release](https://akamas.io/wp-content/uploads/2026/04/akamas-insights-release-analyze-recommend-apply-loop.png-2048x336.png)

Workflow diagram showing the continuous optimization cycle with Akamas Insights: deploy, analyze telemetry, generate AI-powered recommendations, human review, and apply to next release

Every time a new version of the “menu” is released, Akamas Insight performs a deep analysis of the previous cycle’s telemetry. It then generates intelligent, evidence-based recommendations tailored to the Tuning Profile. These recommendations are presented to the Developers for review. This is the “aha!” moment: the developer sees exactly how their code performs and can approve the new configuration with one click, knowing it aligns with the safety standards set by the SRE. This ensures that resource efficiency is balanced against performance and reliability within a controlled, predictable workflow: **Release, Analyze, Recommend, and Apply**. By connecting directly to the telemetry data teams already have, Akamas Insights turns the release cycle into a self-improving loop of continuous harmony.

## The Michelin Standard of Engineering

This transition allows the Platform Team to fulfill its true potential. Instead of being the “Manager” who spends all night mediating between the “Owner” and the “Chef”, they become the architects of a system that empowers Developers to be efficient by default. This creates a culture of “Unit Economics”, where the cost of a transaction is as much a part of the performance metric as latency or error rate. When everyone understands that a 10-millisecond improvement in code efficiency translates directly to a specific reduction in the cloud bill, the silos finally begin to crumble.

The beauty of this harmony is that it liberates the human talent within the organization. Engineers who used to spend their weeks manually tweaking YAML files and arguing over resource limits are suddenly free to innovate. The FinOps team moves from being the “budget police” to being strategic advisors who can predict how a new product launch will impact the bottom line with surgical precision. The SREs can sleep through the night, knowing their clusters are not just “big enough”, but “accurate enough”.

As we look toward the future of cloud-native technology, the goal is no longer just “getting to the cloud”. We are already there. The goal is to live in the cloud with grace and precision. We must move beyond the blunt instruments of the past and embrace a more sophisticated way of managing our digital kitchens. The “Michelin Star” of engineering is awarded to those who can deliver a flawless user experience with the elegance of a perfectly managed budget. By using intelligent, full-stack insights to bridge the gap between our teams, we stop guessing, stop wasting, and finally start cooking with fire.


[Published by DZONE](https://dzone.com/articles/observability-otel-fluentbit)

When we design observability pipelines for modern cloud environments, we implicitly rely on a set of luxurious guarantees: limitless bandwidth, highly available networks, practically infinite storage, and abundant computing power. But when you move these workloads to the edge, think of a maritime vessel navigating the mid-Atlantic or a remote wind turbine, those guarantees vanish. Edge environments are constrained by intermittent connectivity, severe limits on CPU and RAM, and a lack of persistent storage guarantees. You simply cannot run a full, traditional observability stack locally, nor can you stream everything to the cloud without exhausting limited satellite bandwidth.

The engineering challenge becomes clear: how do we build a pipeline that reliably captures traces, metrics, and logs, survives unpredictable network outages, and perfectly correlates signals without saturating edge constraints? A highly compelling, production-realistic solution to this problem was showcased for KubeCon EU 2026, demonstrating a fully correlated observability pipeline built for constrained edge environments using [OpenTelemetry](https://dzone.com/articles/opentelemetry-ending-era-of-fragmented-visibility) and Fluent Bit. You can explore the complete implementation in the [graz-dev/observability-on-edge](https://github.com/graz-dev/observability-on-edge) repository.

This article dives deep into the architecture, the technical trade-offs, and the specific configurations required to make observability work reliably when the network itself is your biggest enemy.

## Taming the Bandwidth With Tail-Based Sampling

The fundamental problem with [distributed tracing](https://dzone.com/articles/distributed-tracing-a-full-guide) is the sheer volume of data it generates. A single HTTP request traversing various middleware and downstream services can easily produce up to 50 individual spans. At a relatively modest load of ten requests per second, you are suddenly dealing with hundreds of gigabytes of trace data over a year. In a cloud environment, you might simply scale up your storage. On a maritime vessel connected via an expensive, low-bandwidth satellite link, sending all of this is economically and technically impossible.

To solve this, we must aggressively sample the data, dropping the noise and keeping only what is actionable. The most naive approach is **head-based sampling**, where the system makes a keep-or-drop decision at the very first span of a trace. While head-based sampling adds almost no computational latency, it is entirely blind to the outcome of the request. If you decide to drop a trace at its inception, and that request subsequently fails or experiences a massive latency spike, that crucial diagnostic information is lost forever. In edge deployments where errors might be rare but highly critical, this is unacceptable.

The solution is **tail-based sampling**. By buffering all spans of a trace in memory within the OpenTelemetry Collector, the pipeline waits for the trace to fully complete before making a decision based on the final outcome. In this implementation, the tail-sampling policy is strictly configured to keep 100% of traces that contain an error, and 100% of traces where the duration exceeds 200 milliseconds. Normal, fast, successful traces are discarded entirely. The observed result is a massive reduction in bandwidth overhead, dropping roughly 80% of all spans before they are ever exported over the network.

To make this bandwidth reduction truly effective, we must apply the same philosophy to our logs. If we filter traces but send every single log line, we defeat the purpose. For log processing, Fluent Bit runs as a DaemonSet on the edge nodes, tailing container logs. Rather than using Fluent Bit's native grep filters, which struggle with complex multi-field conditional logic, a custom Lua filter is injected into the pipeline. This Lua script precisely mirrors the OpenTelemetry tail-sampling criteria, evaluating each log record and keeping only those with an error level or a duration exceeding 200 milliseconds. By performing this logic at the absolute edge, before the log data even leaves the node, the pipeline drops approximately 86% of log volume at the source, preventing unnecessary network I/O.

## Persistent Queuing for Intermittent Connectivity

Aggressive sampling solves the bandwidth issue during steady-state operations, but what happens when the satellite link inevitably fails? If the OpenTelemetry Collector cannot reach the central hub, it will quickly exhaust its in-memory retry buffers, and all telemetry accumulated during the outage will be permanently lost.

To survive these disconnections, the pipeline implements a file-backed persistent queue utilizing the `file_storage` extension within the OpenTelemetry Collector. This provides a bbolt (BoltDB) key-value store directly on the edge node's local disk. When the exporter's sending queue is configured to use this file storage, outgoing telemetry batches are serialized and written safely to the crash-safe bbolt database before dispatch. These items remain safely on disk until the collector receives a successful acknowledgment from the remote endpoints.

Configuring this queue correctly requires understanding a critical nuance of the OpenTelemetry Collector's internal architecture regarding consumers. By default, the exporter uses four concurrent consumer goroutines to claim batches from the queue. Because the processing pipeline generally produces batches relatively slowly at edge traffic volumes, these four consumers will claim batches almost instantly, holding them in their in-memory retry buffers rather than leaving them in the bbolt queue.

Consequently, your queue depth metrics will deceptively report zero even during an active network outage, blinding you to the growing backlog. By deliberately setting the `num_consumers` configuration to exactly one, only a single batch is ever held in flight in memory. All subsequent batches safely queue in bbolt, allowing the metric to accurately reflect the growing backlog during an outage.

## The Reconnection and the Time Travel Problem

When the satellite link is eventually restored, the real chaos begins. The OpenTelemetry Collector detects the restored connection and immediately begins draining the accumulated bbolt queue at maximum network speed. However, the data stored in this queue possesses timestamps from minutes or hours in the past. We are essentially attempting to push historical, out-of-order data into our central observability backends. Different backends handle this "time travel" problem differently. Jaeger, our distributed trace storage, handles it gracefully by design. Its storage model is append-only, possessing no concept of out-of-order rejection. Traces originating from the failure window simply appear in the user interface precisely where they belong chronologically.

[Loki](https://dzone.com/articles/grafana-loki-fundamentals-and-architecture), handling our logs, is much stricter. By default, Loki expects log entries for a given stream to arrive in roughly chronological order, and it will forcefully reject significantly older timestamps with HTTP 400 errors. If left unconfigured, the OpenTelemetry Collector would receive these errors continuously upon queue drain, leading to the permanent loss of all logs generated during the outage. To prevent this disaster, we must explicitly configure `unordered_writes: true` in the Loki settings. This crucial parameter disables the strict per-stream ordering requirement, allowing the massive burst of queued, historical log entries to be ingested successfully.

Metrics ingestion presents an even harsher reality. In this architecture, metrics are exported to Prometheus using the `prometheusremotewrite` exporter. Unlike logs and traces, the OpenTelemetry Collector library supporting this exporter lacks support for the file-backed bbolt queue, leaving the metrics queue strictly in-memory.

Furthermore, when the link restores, any old metrics held in memory are sent to Prometheus, but Prometheus natively rejects out-of-order samples. While there are alternative protocols like OTLP HTTP for metrics, utilizing OTLP for Prometheus ingestion results in aggressive HTTP 400 rejections for out-of-order data. This causes the exporter to retry indefinitely, permanently blocking the queue and grinding the entire metrics pipeline to a halt.

It is crucial to note that this specific limitation — the inability to use file-backed queues for metrics — is a known library constraint tied directly to the OpenTelemetry Collector versions (v0.95 and v0.96) utilized in this repository. Because these specific builds do not support `sending_queue.storage: file_storage` for the Prometheus remote-write exporter, the architecture is forced to keep the metrics queue in RAM. The architectural decision here is a deliberate and calculated engineering trade-off: by using the `prometheusremotewrite` exporter targeting the remote-write endpoint, Prometheus silently skips the out-of-order samples by returning an HTTP 204 status with zero written. The pipeline queue drains cleanly and unblocks, but the metric data generated during the actual outage window is intentionally sacrificed. At the edge, maintaining pipeline integrity is often prioritized over absolute metric continuity.

## Achieving Deterministic Signal Correlation

An observability stack is only as valuable as its ability to correlate signals. A latency spike on a dashboard must lead seamlessly to the exact distributed trace, which must seamlessly transition to the specific log lines emitted during that exact request. In this edge architecture, achieving perfect correlation is not reliant on best-effort timestamp matching; it is structurally guaranteed.

The process begins in the application code, which extracts the OpenTelemetry `trace_id` and `span_id`  
from the context of every incoming HTTP request and structurally injects them into every log line via a JSON logger. Because the Fluent Bit Lua filter and the OpenTelemetry tail-sampling processor utilize the exact same logic, we achieve a deterministic alignment. Every trace that survives the sampler will have a corresponding log line surviving the Lua filter, and conversely, no log line will exist without a parent trace. There are no orphaned traces and no orphaned logs. To link our high-level metrics directly to these traces, the architecture employs the OpenTelemetry `spanmetrics` connector. This connector reads the sampled spans and generates Prometheus histogram metrics regarding request rates and latencies. Crucially, it attaches an exemplar to each histogram bucket, a sparse metadata annotation carrying the specific `trace_id` that contributed to that latency measurement.

The placement of this connector is paramount. In the configuration pipeline, `spanmetrics` is wired strictly after the tail sampling processor. Because it runs post-sampling, every single `trace_id` it embeds into a metric exemplar is absolutely guaranteed to have survived the sampling process and exist in Jaeger. When operators view their Grafana dashboards, they see diamond markers on their latency graphs representing these exemplars; clicking a marker reliably drops them directly into the exact failing trace with zero dead links.

## Performance Testing

To prove this architecture works under realistic conditions, the project doesn't just send a few manual requests. Instead, it employs the k6 Operator, a Kubernetes-native load test runner, to generate a continuous, high-volume telemetry stream from within the cluster hub. The load generator utilizes a custom `TestRun` resource that spins up 500 virtual users over a 40-minute period.

This performance test follows a deliberate ramp-up profile: it scales from zero to 100 virtual users in the first 30 seconds, climbs to 250 in the next 30 seconds, reaches 500 at the one-minute mark, and sustains that peak for a 40-minute steady state. At its peak, this setup generates approximately 2,500 spans per second. The traffic is intelligently distributed across four distinct API endpoints simulating a vessel's systems, each with specific latency and error profiles, perfectly exercising the tail-sampling and Lua filtering logic.

While the k6 load test validates the pipeline's throughput, the underlying reality is that rigorous performance testing was essential for a much more critical goal: drastically minimizing the collector's resource footprint. In constrained edge environments, every megabyte of RAM and CPU cycle consumed by observability is stolen directly from the primary application workloads. Through an extensive, automated performance tuning campaign, we analyzed the complex interactions between the Go runtime and the collector's internal processors. The findings revealed that optimizing an edge node requires surgically tuning both the OTel configuration and the underlying Go environment rather than simply guessing at limits. By methodically testing various permutations, we discovered the exact "sweet spot" to maximize performance while shrinking the footprint.

The most impactful findings from these tests led to highly specific internal calibrations. For example, the `memory_limiter` processor was precisely tuned to enforce a soft limit of 320 MiB and a hard limit of 400 MiB. This was paired with a batch processor rigorously configured to accumulate exactly 512 spans or wait for a maximum 5-second timeout. Furthermore, these tests demonstrated that throttling the exporter queue to a single consumer (`num_consumers: 1`) was critical. It not only provided accurate backpressure metrics during a simulated satellite outage but structurally prevented the Go runtime's garbage collector from thrashing when the connection was restored and massive historical queues suddenly drained.

The results of this optimization campaign are striking. Stripped of the bloated default components via the OpenTelemetry Collector Builder, the resulting 30 MB binary operates seamlessly under intense pressure. It continuously processes thousands of spans per second while consuming merely 1% to 5% of a single CPU core and hovering predictably between 80 MiB and 150 MiB of active memory. This definitively proves that with proper performance testing and exact Go runtime calibrations, you do not have to choose between rich telemetry and edge node stability.

## Validating Resilience Through Network Chaos

Proving that resilience actually works requires simulating harsh physical realities within a controlled Kubernetes environment. Relying on high-level Kubernetes NetworkPolicies is insufficient for this testing, as they do not provide the surgical, instantaneous, and reversible IP-layer control needed to simulate an abrupt satellite drop. Instead, the project utilizes a privileged DaemonSet running `netshoot`, a network debugging container. Operating in the host network namespace, this pod can directly manipulate the edge node's kernel routing rules using `iptables`. A dedicated chaos script surgically inserts DROP rules into the FORWARD chain, specifically targeting the outbound ports for Jaeger, Loki, and Prometheus.

A critical detail in this simulation is the behavior of the Linux kernel's connection tracking framework, `conntrack`. Modern kernels maintain state for established TCP connections, allowing them to bypass newly inserted DROP rules. If you apply an `iptables` drop rule without further action, existing gRPC connections between the collector and the hub will simply continue to flow unaffected. The chaos script explicitly executes a `conntrack` flush command targeting the collector's IP address. This violently terminates the established states, forcing the client to initiate a new TCP handshake, which is immediately blocked by the new rules. This accurately triggers the failure state: the OpenTelemetry exporter begins failing, batches begin piling up in the bbolt database, and the queue depth metrics steadily climb. Removing the rules simulates link restoration, triggering the massive, satisfying spike in export throughput as the resilient queue drains historical data into the backends.

## Conclusion

Observability at the edge forces engineering teams to abandon the comfortable defaults of cloud-native computing. We cannot afford to transmit every metric, log line, and trace span. By combining aggressive tail-based and source-based sampling, highly localized persistent queuing, out-of-order gap-filling configurations, and meticulous correlation through exemplars, it is completely possible to maintain deep, actionable visibility into remote, constrained environments.

The implementation presented in the [graz-dev/observability-on-edge](https://github.com/graz-dev/observability-on-edge) repository serves as an example of these techniques. It proves that with strict resource management and a deep understanding of network behavior, robust edge observability is not just a theoretical concept, but a highly achievable engineering reality.

Making cloud native easier for the humans inside it. OSS Contributor, Co-Chair @ CNCF TAG DevEx, DevRel Engineer at Akamas, Kubernetes Blog Maintainer.

Graziano Casto — OSS Contributor & Co-Chair @ CNCF TAG DevEx

Graziano Casto

OSS Contributor & Builder (K8s), Co-Chair of the CNCF TAG Developer Experience, DevRel Engineer at Akamas, Kubernetes Blog Maintainer.

DevRel Engineer

Akamas

Making cloud native easier
for the humans inside it.

Latest Activity

Featured Articles

Upcoming Events

Making cloud native easierfor the humans inside it.

Latest Activity

Featured Articles

Upcoming Events

Making cloud native easier
for the humans inside it.