Troubleshooting

Common Issues

Workload is idle but not being paused

Check 1: Is dry run enabled?

kubectl get managedworkload my-api -n staging -o jsonpath='{.spec.dryRun}'

If true, the operator evaluates but doesn't act. Set to false to enable.

Check 2: Are all signals confirming?

kubectl describe managedworkload my-api -n staging

Look at events for signal evaluation results. If any signal denies, idle detection resets.

Check 3: Is the grace period still running?

Check the idle signal metric:

# 3 = InGracePeriod, 4 = Idle
kubectl get --raw /metrics | grep hybernate_idle_signal_result

Check 4: Is the prediction engine active?

kubectl get managedworkload my-api -n staging -o jsonpath='{.status.prediction}'

If dailyPhase is Observing, the engine hasn't collected enough data yet (needs 24+ hours).

If it has been Observing for well over 24 hours, check whether the engine is being fed at all:

kubectl get managedworkload my-api -n staging \
  -o jsonpath='{.status.conditions[?(@.type=="MetricsAvailable")]}'

MetricsAvailable=False means the operator can't read the workload's CPU usage, so the forecast isn't learning:

• NoPodMetrics: the target has replicas, but the Metrics API reports no pods for it. Check that metrics-server is installed (kubectl top pods -n staging) and that the target's pods are running.
• MetricsUnavailable: the Metrics API itself failed. The condition message has the error; a missing metrics.k8s.io API means metrics-server isn't installed.

A target scaled to zero replicas is not an error: it's recorded as zero demand.

Workload keeps cycling between paused and running

This usually means idle detection triggers pause, then auto-resume immediately detects "not idle" (because paused workloads have zero CPU, which is below threshold, but the workload has no pods to measure).

Fix: Ensure your Prometheus signals check for actual traffic, not just CPU:

idlePolicy:
  signals:
    - source: prometheus
      promQL: 'rate(http_requests_total{service="my-api"}[10m]) == 0'

Prediction confidence stays at 0

The confidence scorer needs a full 24-hour window of data before reporting. Wait 24+ hours after creating the ManagedWorkload.

Also check that metrics-server is running and returning data:

kubectl top pods -n staging

Target not found

kubectl get managedworkload my-api -n staging -o jsonpath='{.status.conditions}'

If you see a Degraded condition with "target not found":

• Verify the target exists: kubectl get deployment my-api -n staging
• Check that target.kind matches (Deployment vs StatefulSet)
• Ensure the ManagedWorkload is in the same namespace as the target

Duplicate target error

Only one ManagedWorkload can manage a given workload. Check for duplicates:

kubectl get managedworkloads -n staging -o jsonpath='{range .items[*]}{.metadata.name}: {.spec.target.name}{"\n"}{end}'

Cost data shows $0.00

• Cost accumulation requires metrics-server data. Check kubectl top pods.
• On day 1 of the month, estimatedMonthlyCost shows "pending" until day 2.

Debug Checklist

1. Operator running? kubectl get pods -n hybernate-system
2. CRDs installed? kubectl get crd managedworkloads.hybernate.io
3. Metrics server running? kubectl top nodes
4. RBAC correct? kubectl auth can-i get pods --as=system:serviceaccount:hybernate-system:hybernate-controller-manager
5. Events? kubectl describe managedworkload <name> -n <ns>
6. Logs? kubectl logs -n hybernate-system deployment/hybernate-controller-manager
7. Status? kubectl get managedworkload <name> -n <ns> -o yaml

Getting Help

If you've gone through this checklist and still have issues:

1. Check the GitHub Issues for known problems
2. Open a new issue with:
3. Operator version and Kubernetes version
4. ManagedWorkload YAML (sanitized)
5. Relevant operator logs
6. Output of kubectl describe managedworkload