lastwords

Memory snapshots before OOM kills, and logs and core dumps from crashes, for Kubernetes.

lastwords runs on every node. When a container gets close to its memory limit, it records what is using the memory. When a container dies, it saves the logs of that instance, the pod's events and any core dump. Everything stays on the node (or in S3), and you can read it with kubectl lastwords or from an AI agent over MCP.

helm install lastwords oci://ghcr.io/arvo-ai/charts/lastwords -n lastwords --create-namespace

The problem

When you look at a pod that died hours ago, most of the evidence is gone:

• OOMKilled leaves no heap dump. The kernel kills the process with SIGKILL. -XX:+HeapDumpOnOutOfMemoryError never fires, and Go, Node and Python get no warning either.
• kubectl logs --previous only keeps one restart. After a few crash-loop restarts, the logs from the first failure are gone.
• Events expire after one hour (--event-ttl, default 1h).
• Core dumps go nowhere. core_pattern is set per node, so crashing containers either dump nothing or dump to a path inside a container that is about to be deleted.

None of it can be recovered later. It has to be collected while the container is alive or right after it dies, which is what lastwords does.

What it captures

Every capture comes with a summary.md that says what happened in plain words:

# checkout-5498f94695-526hm/checkout was OOMKilled at 16:51:38 UTC (limit 384.0 MiB); 1 snapshot taken before the kill

## What lastwords found
- Memory climbed steadily for 1m1s, the pattern of a leak or an unbounded cache. Raising the limit would only delay the next kill.
- Before the kill (67% of limit): Largest application classes on the heap:
  com.example.checkout.CheckoutService$CartSnapshot: 2860 instances ...
- Before the kill (67% of limit): One class dominates the Java heap: byte[] holds 179.2 MiB (97%).

# order-consumer-76d5fcb94-8vm4p/consumer crashed at 03:09:30 UTC: Error, exit code 2

## What lastwords found
- Crashed in main.registerDiscount at /src/main.go:19 (assignment to entry in nil map).

Use it

kubectl lastwords list                      # everything from the last 7 days
kubectl lastwords list -n shop --kind oomkill
kubectl l

... [View full README on GitHub](https://github.com/arvo-ai/lastwords#readme)