
[Published by DZONE](https://dzone.com/articles/infrastructure-crossplane-kube-green)

We were all sold a compelling vision of cloud computing: one filled with agility, endless scalability, and remarkable cost savings. Yet, for many of us in the trenches, the daily reality looks quite different. We find ourselves wrestling with an infrastructure model built on long-lived, static environments for development, testing, and staging. This old way of working has quietly become a massive drain on our resources, creating financial waste, operational headaches, and a growing list of security and environmental debts.

This isn't just one problem; it's a vicious cycle. The friction in our daily operations directly fuels the financial, security, and environmental burdens. To break free, we need more than just a new tool; we need to fundamentally rethink how we provision, manage, and consume infrastructure.

## The High Cost of 'Always-On'

The most obvious symptom of this broken model is the money it wastes. It's not a small number; research suggests a staggering [30-45% of all cloud spending is wasted on idle or over-provisioned resources](https://42on.com/the-hidden-costs-of-cloud-waste-how-to-optimize-your-cloud-resources-and-reduce-environmental-impact/). This isn't surprising when you think about it. Our staging, UAT, and dev environments are built to handle peak loads, but they spend most of their lives (nights, weekends, holidays) doing absolutely nothing. This leads to an abysmal average CPU utilization of just 12-15%, a clear sign that something is deeply inefficient. This problem is made worse by "cloud sprawl", where new resources pop up uncontrolled, increasing costs, security holes, and complexity.

This financial drain is a direct result of operational friction. Many of us are familiar with the "ticket-driven" culture: a developer needs an environment, so they file a ticket and wait. And wait. This manual, slow process creates bottlenecks that grind the entire development lifecycle to a halt. In a world of complex microservices, setting these environments up by hand isn't just slow; it's a recipe for errors. This model encourages us to keep these costly, complex environments running indefinitely, because tearing them down and rebuilding them is just too painful.

This persistence creates a huge security risk. Over time, these static environments suffer from "configuration drift." Manual tweaks, untracked updates, and inconsistent patching mean they no longer match production or even each other. They become, as one source aptly put it, a ["hive of duplicated data, exploitable vulnerabilities, and misconfigurations"](https://www.quali.com/glossary/ephemeral-environments/). Each one is a persistent attack surface, a liability that only grows with time.

## The Unseen Environmental Toll

The money we waste on idle resources is just the tip of the iceberg. Under the surface lies a massive, often ignored, environmental cost. Every idle virtual machine or database isn't just a line item on a bill; it's a physical server in a data center, burning electricity and pumping out CO2 for no reason. Data centers already consume about [1% of the world's electricity](https://www.researchgate.net/publication/390569272_The_Environmental_Impact_of_Cloud_Computing_Sustainability_in_the_Cloud), and that number is only going up.

To put this in perspective, a 2019 study found that training a single large AI model can produce the same carbon emissions as [more than 300 cross-country flights](https://www.forbes.com/councils/forbestechcouncil/2025/02/20/the-unsung-mechanics-how-ai-and-cloud-integration-drive-the-engine-of-sustainability/). This is the energy cost of our digital world. Beyond emissions, there's e-waste from constant hardware upgrades and the immense amount of water used for cooling. While cloud providers are more efficient than on-premise data centers, the core problem remains: we, the customers, are provisioning resources that sit idle, driving unnecessary environmental impact.

Let's make this real. Imagine a team of 10 engineers with a single, static staging environment. It runs 24/7, but they only use it for about 160 hours a month. For the other 560 hours, nearly 78% of the time, it's idle but still racking up costs. Using the CO2 calculator from the [kube-green project](https://kube-green.dev/), which assumes that a single pod produces about [11 kg of CO2eq per year](https://kube-green.dev/docs/FAQ/#how-much-co2-is-produced-by-a-pod), we can see the impact. If that staging environment runs 20 pods, it generates 220 kg of CO2eq annually. By simply shutting it down when it's idle, the team could prevent over 170 kg of CO2eq emissions. Now, scale that across an entire organization. The waste is enormous.

## The Ephemeral Revolution

This is where a new idea comes in: the ephemeral environment. It’s a complete shift in thinking. Instead of permanent, long-lived infrastructure, we move to on-demand, temporary workspaces that directly combat waste and friction. An ephemeral environment is a short-lived, isolated, and complete copy of your application, spun up for a specific task and automatically destroyed when you're done.

This approach is built on a few key ideas. First, it's **on-demand and automated**. Environments are created by events in your workflow, like opening a pull request, not by filing a ticket. What once took days now takes minutes. Second, each environment is **completely isolated**. This means no more "who broke the staging server?" A developer can test a new feature, run a database migration, or experiment with a dependency without affecting anyone else. This enables true parallel development.

Crucially, these are not just simple "preview environments". A true ephemeral environment achieves **production-parity**. It’s a full-fidelity clone of your production stack: backend services, databases, configurations, everything. This is how you kill the "it works on my machine" problem for good. Finally, these environments are both **shareable and disposable**. You get a unique URL to share with stakeholders for feedback on a live, running feature. When the pull request is merged, the entire environment is wiped away as if it never existed. This disposable nature is the key to its efficiency.

Adopting this model has a transformative impact. It dramatically boosts developer velocity by removing bottlenecks. Some organizations have seen [delivery times improve by up to 35%](https://ephemeralenvironments.io/) and overall velocity increase by as much as 10x. It fosters better collaboration because everyone, from product managers to QA, can interact with a tangible, running feature, not just mockups. This "shift-left" approach to testing also means you catch bugs earlier, significantly improving code quality and reducing the risk of production failures. The old static staging environment is a relic of an old era: costly, slow, and risky. The ephemeral model is its opposite: on-demand, fast, safe, and efficient.

## The Technology Pillars: Crossplane and kube-green

This vision of a fully ephemeral cloud requires the right technology. You need to be able to programmatically control not just your application containers, but the entire infrastructure ecosystem: databases, networks, storage, and more. This is where two key open-source projects come into play: Crossplane and kube-green.

Crossplane is the heart of this architecture. It’s a universal control plane that extends the Kubernetes API to manage any cloud resource, not just containers. This represents a profound shift from traditional [Infrastructure as Code (IaC)](https://dzone.com/articles/what-is-infrastructure-as-code) tools like Terraform to a new model: [Infrastructure as Data (IaD)](https://medium.com/@swarupdonepudi/infrastructure-as-data-iad-coincidence-2ca6c70f896e).

With traditional IaC, you run a command like terraform apply to make a one-time change. With Crossplane, you declare the desired state of your infrastructure in a YAML file and submit it to the Kubernetes API. From then on, Crossplane's controllers work continuously to ensure reality matches your declaration. If anything drifts, Crossplane automatically corrects it. This "self-healing" capability is a game-changer.

It works through a clever, layered architecture. **Providers** teach Crossplane how to talk to a specific API, like AWS or GCP. **Managed Resources** are the building blocks, representing a single piece of infrastructure like an S3 bucket or an RDS database. But the real magic is in **Compositions**. A platform team can use a Composition to bundle all the necessary Managed Resources into a single, high-level abstraction, like a PostgreSQLInstance. This blueprint can embed all the company's best practices for security, networking, and compliance.

Developers don't need to worry about that complexity. They simply request what they need by creating a **Claim**: a simple YAML asking for a PostgreSQLInstance for example. Crossplane sees the claim and provisions the entire, production-ready stack automatically. This is true developer self-service, and it's exactly what's needed for ephemeral environments. When a pull request is opened, a CI/CD pipeline can create a single Claim that tells Crossplane to spin up a dedicated database, message queue, and everything else the application needs. When the PR is closed, deleting that one Claim tears it all down.

While Crossplane manages the existence of the environment, [kube-green](https://kube-green.dev/) handles the efficiency within that existence. An ephemeral environment might live for a few days, but developers aren't working on it 24/7. kube-green is a simple, brilliant utility that solves this "last mile" problem by automatically hibernating resources on a schedule.

It operates as a Kubernetes operator that you configure with a **SleepInfo** resource in each namespace. You can tell it to shut things down at 8 PM on weekdays and wake them up at 8 AM, for example. When the sleep time hits, kube-green scales down your Deployments and suspends your CronJobs. When the wake-up time arrives, it brings everything back to its original state. This simple action can lead to huge savings. Adopters have reported [30-40% reductions in cloud costs](https://kube-green.dev/docs/adopters/) for non-production clusters. And because it reduces energy consumption, it has a direct, positive impact on your carbon footprint.

## The Synthesis: A New Blueprint for Efficiency

When you combine Crossplane's strategic, full-stack orchestration with kube-green's tactical, time-based optimization, you get something truly powerful. This synergy creates a multi-layered system of efficiency that aligns your infrastructure cost and carbon footprint with the actual, value-generating work of your development teams.

The workflow, driven by GitOps tools like ArgoCD, is beautifully simple. A developer opens a pull request. This triggers a CI pipeline that generates the manifests for the new environment, including a Crossplane Claim for the infrastructure and a kube-green SleepInfo for the schedule. These manifests are committed to a Git repository. ArgoCD sees the new files and applies them to your management cluster. Crossplane springs into action, provisioning the entire infrastructure stack, perhaps in a lightweight vCluster for perfect isolation. Once the infrastructure is ready, ArgoCD deploys the application code. When the PR is merged or closed, a CI job removes the manifests from Git, and the entire environment vanishes.

This creates a two-tiered optimization model. **Macro-optimization** from Crossplane ensures you only pay for infrastructure for the few days a PR is active, not for a permanent staging environment. **Micro-optimization** from kube-green then eliminates waste within that short lifespan by hibernating resources during non-working hours.

This points to an exciting point: a "deep sleep" model. Using kube-green's ability to patch any Kubernetes resource, you could have it not only scale down pods but also patch the Crossplane Claim itself, for example, telling it to scale a database cluster down to a single, low-cost node overnight. This would extend time-based optimization to the entire stateful infrastructure stack, achieving the ultimate form of on-demand efficiency.

## Getting Started on Your Journey

While this vision of a fully automated, per-PR ephemeral world is the goal, you don't have to build it all at once. The journey can start with a single, high-impact step. To make these concepts tangible, I've put together a [hands-on tutorial](https://github.com/graz-dev/crossplane-ephemeral-environments) that focuses on a pragmatic starting point: creating on-demand infrastructure that's only active when you need it.

The tutorial walks you through using Crossplane to provision an AWS EKS cluster and then using kube-green to put it on a sleep/wake schedule. It's a direct, practical way to tackle the biggest source of waste: idle resources.

If this vision for a more efficient cloud resonates with you, the best way to understand it is to build it. I invite you to check out the tutorial, [Ephemeral Cloud Resources with Crossplane and kube-green](https://github.com/graz-dev/crossplane-ephemeral-environments). Dive in, experiment with the code, and see the benefits for yourself. If you find it valuable, please give the repository a star on GitHub and share this article with your network.

## A Strategic Outlook for the Future

The shift to an ephemeral cloud model is more than a technical upgrade; it's a strategic evolution. It allows us to move from passively managing static resources to actively orchestrating dynamic, value-generating capabilities. The future of this model is even more exciting, driven by AI-powered optimization and the formalization of GreenOps.

New AI tools like StormForge and CAST AI are emerging that can analyze workloads to automatically tune resource configurations and predict demand. Imagine an AI that could predict a PR's review cycle to allocate resources more intelligently or manage energy use based on the real-time carbon intensity of the power grid.

This model is also a foundational practice for GreenOps, the discipline of bringing environmental accountability to cloud operations. As sustainability becomes a key business metric, the ability to measure and minimize the carbon footprint of software development will be a major differentiator.

For technology leaders, the path forward is clear. Treat your internal infrastructure as a product and invest in a platform team to build out this capability. Foster a culture of accountability by making cost and carbon data transparent to your engineering teams. And start small. Pick a single service and use it as a pilot project to prove the value and build expertise.

The combination of Crossplane and kube-green offers a powerful and pragmatic toolkit to begin this journey. By adopting this new blueprint, you can accelerate innovation while dramatically reducing your financial and environmental costs.


[Published by Cloud Native PE Community](https://cloudnativeplatforms.com/blog/kcd-nyc-platform-engineering-and-observability-roundtable/)

> There was a great energy at KCD New York this year, and for Graziano Casto, a personal highlight was leading a roundtable on observability. It was a fascinating discussion that really got him thinking about the challenges we’re all facing in the platform engineering space. Here is Graziano’s recap of the key problems and promising ideas that came up.

### Introduction

It was an absolute pleasure recently to moderate a roundtable at [KCD New York](https://community.cncf.io/events/details/cncf-kcd-new-york-presents-kcd-new-york-2025/), diving deep into the fascinating (and let’s be honest, sometimes frustrating) world where **Platform Engineering meets Observability**. As my first time moderating a roundtable, I was genuinely thrilled by the energy and candid participation from everyone in the room. A huge thank you to all the participants: [Michel Murabito](https://www.linkedin.com/in/mich-murabito/), [Colin Lacy](https://www.linkedin.com/in/colinjlacy/), [Tiara Sykes](https://www.linkedin.com/in/tiara-sykes/), [Andrew Espira](https://www.linkedin.com/in/andrew-espira/), [Mariia Rudenko](https://www.linkedin.com/in/mariia-r-748931163/), [Aderianna Williams](https://www.linkedin.com/in/at-williams/), [Marino Wijay](https://www.linkedin.com/in/mwijay/) and [Maria Ashby](https://www.linkedin.com/in/maria-ashby/) whose insights made this discussion truly invaluable. We had an incredibly insightful exchange, and I walked away with some serious food for thought.

![](../../assets/newsletter/12-kcd-new-york/photo-4.jpeg)

We kicked off by acknowledging a universal truth in today’s cloud-native landscape: managing a full observability stack often feels like trying to hit a moving target. The more we aim to observe, the more inherent complexity seems to creep in. We continuously pile on tools, data, and dashboards–be it metrics, traces, logs, or profiling – and suddenly, we’re swimming in a sea of cognitive load, entropy, and quite often, plain old confusion. So, instead of me listing the common headaches, I threw it open to the room: “When you think about managing a full observability stack, across logs, metrics, traces, and so on, what are your biggest pain points? If you had to name the biggest challenge your team is facing with observability right now, what would it be?”

The responses flowed freely, revealing a shared understanding that while observability promises clarity, its real-world implementation often introduces its own unique set of challenges. And, quite organically, our conversation drifted into the exciting (and slightly unsettling) realm of Generative AI, specifically discussing how Large Language Models (LLMs) can be synergistically integrated within platforms to serve as enablers in resolving some of these very challenges.

### The Observability Headaches: Where Do We Feel the Pinch?

One of the loudest points of contention was the persistent struggle to correlate telemetry data with the actual services generating them, and the broader challenge of **telemetry data correlation**. It’s like having all the pieces of a complex puzzle but no clear idea how they fit together. You might spot a spike in CPU utilization – a metric – but then you’re left guessing which microservice is the culprit. Then begins the detective work: diving into logs to pinpoint an error, and finally tracing requests to understand the flow. The fundamental problem is that these critical data points often reside in disparate systems, use inconsistent identifiers, and demand a significant amount of manual effort and intuition to connect the dots. This fragmentation makes it incredibly difficult to quickly identify the root cause of a problem when seconds count.

Adding to this complexity is the sheer volume of alerts and the difficulty in correlating them with the actual underlying problem. We’ve all experienced it: a dozen alerts fire simultaneously, each pointing to a symptom, yet none clearly indicating the core issue. This leads to what’s known as **alert fatigue**, resulting in missed critical incidents, wasted time triaging false positives, and ultimately, a palpable loss of trust in the alerting system itself. The challenge isn’t merely about receiving notifications; it’s about receiving meaningful alerts that directly pinpoint the underlying problem, not just its outward manifestations.

Furthermore, a significant unspoken burden that often comes with observability is the **cost** of both creating and maintaining the entire observability stack. From licensing fees for proprietary tools to the infrastructure costs of storing massive volumes of telemetry data and the operational overhead of managing these complex systems, the financial outlay can be substantial. This constant investment of resources, both human and monetary, can become a major pain point, often weighing heavily on budget decisions and resource allocation.

Then there’s the pervasive issue of **making insights accessible** and visualizing them in a way that provides the right insight to the right person. Raw telemetry data, in its unadulterated form, is overwhelming. Different roles within an organization – SREs, developers, product managers – need distinct views and varying levels of detail. A developer might require granular trace data, while a product manager needs high-level business metrics. The constant battle involves creating and maintaining these tailored dashboards and ensuring everyone knows where to find what they need. This often leads to information silos and missed opportunities for proactive improvement.

A recurring theme throughout our discussion was the persistent problem of **siloed teams** and the resulting **lack of standardization**. When different teams adopt disparate observability tools, inconsistent naming conventions, or even varied logging formats, it inevitably creates a fragmented and chaotic landscape. This makes it incredibly challenging to gain a holistic view of the system, collaborate effectively during incidents, and leverage best practices across the entire organization. It’s a classic case of “everyone doing their own thing”, leading to pervasive inefficiencies and increased complexity.

Finally, a crucial point that resonated deeply was the importance of **developer education**. Observability isn’t merely about deploying tools; it’s about cultivating a specific mindset. Developers need to grasp why observability is vital, how to effectively instrument their code, how to interpret telemetry data, and critically, how to leverage observability tools to troubleshoot their applications. This knowledge gap can lead to poorly instrumented services, ignored alerts, and a general underutilization of the powerful observability stacks organizations invest heavily in.

### Internal Developer Platforms: The Unified Solution

So, with these common headaches laid out, how do we begin to alleviate them? This is precisely where the concept of an **Internal Developer Platform (IDP)** steps in as a truly powerful solution, providing a cohesive answer to many of these challenges.

An IDP, at its core, inherently solves the problem of standardization. By providing clear standards and abstractions through “golden paths” for building and deploying applications, it ensures consistency across the organization. However, it’s crucial that these **golden paths** don’t become “golden cages”. A well-designed IDP empowers developers with the necessary autonomy to cover edge cases, allowing them to step outside the perimeter of the provided golden paths when needed for specific requirements. This balance is vital for both consistency and innovation.

Moving beyond standardization, IDPs also play a crucial role in addressing the challenges faced by siloed teams that might be working on different components of the same system and often lack a shared performance baseline. During our discussion, we introduced the concept of leveraging generative models within these platforms. Specifically, the role of Generative AI, particularly **Large Language Models (LLMs)**, in the observability space emerged as a truly futuristic and exciting prospect. The idea is that LLMs can help close the gap between users and telemetry data. Imagine being able to ask natural language questions like, “Why is our checkout service slow right now?” and have an LLM sift through mountains of metrics, logs, and traces to provide a concise, actionable answer. Or, “What were the top 3 errors in our authentication service last night?” and get a summary, perhaps even with links to relevant log lines. These models can also be instrumental in enabling teams to define and compare their telemetry data against customized thresholds, ensuring that the entire system is monitored according to a collectively defined baseline, fostering a shared understanding of system health.

From here, we delved into how these models further enhance the transparency and clarity of insights. LLMs, integrated within the IDP, can analyze vast amounts of telemetry data and provide various stakeholders with personalized insights and alerts. This capability opens the door to entirely new interfaces beyond the traditional dashboards, making complex operational data more accessible and actionable for different roles. Unfortunately, we didn’t have the opportunity to delve deeper into the intricate topic of telemetry data correlation during the roundtable, but I have written an article that explores this topic further, which you can find [here](https://www.linkedin.com/pulse/9-serving-observability-first-dish-graziano-casto-05rhf).

### The Open Question: Balancing Trust and Cost with Benefits in the LLM Era

However, as with any powerful new technology, the discussion around LLMs quickly led to a critical open question for the community:

**How do we effectively balance the significant benefits that LLMs bring – such as improved automation and deeper insights – against the inherent costs? These costs include not only the economic investment required for these models but also the crucial aspect of trust, both in the accuracy of the results and in entrusting our sensitive data to an LLM, particularly when utilized as a service.**

This is a conversation that needs to continue. As we push the boundaries of what’s possible with AI in operations, we must collectively figure out how to build systems that are not only efficient and intelligent but also fundamentally secure, trustworthy, and cost-effective.

### Wrapping Up

My first moderating experience at KCD New York was an absolute blast, and the insights from the roundtable on Platform Engineering and Observability were truly invaluable. It’s clear that while observability brings its own set of complexities, Internal Developer Platforms offer a robust framework for overcoming these challenges by promoting standardization, providing contextualized insights, and empowering developers. And looking ahead, the potential of LLMs to revolutionize how we interact with our telemetry data is incredibly exciting, even if it comes with some important questions we need to answer as a community.

What are your thoughts on these challenges and solutions? And how do you see the role of LLMs evolving in the observability space, especially concerning the trust and cost trade-offs? Let’s keep the conversation going!


Welcome to the personal website of Graziano Casto.

Graziano Casto 👋🏻

Cloud Native Friday Newsletter

Pods & Platforms

Latest Articles

Kubernetes v1.34: Of Wind & Will (O' WaW)

The Ephemeral Cloud: A New Blueprint for Infrastructure Efficiency With Crossplane and kube-green

From Chaos to Clarity: Navigating Observability in the Platform Engineering Era (and a Dash of AI)

WebAssembly: From Browser Plugin to the Next Universal Runtime

Upcoming Talks