A Kubernetes operator often looks like the sensible default when a team adopts a new infrastructure tool: install it once, declare the desired state, and let the controller handle the rest. For most teams, that is the wrong starting point. An operator adds a control loop and sometimes a new admission dependency to a cluster you already have to keep running. Start with the least privileged installation that can meet the production requirement.
An operator makes a tool’s failure part of cluster operations
The attraction is understandable. A vendor supplies CustomResourceDefinitions (CRDs), a Helm 3 chart, and examples of the custom resources to apply. Argo CD can keep the manifests synchronized. Compared with a page of manual setup instructions, the result looks reproducible. But a reproducible installation is not the same as a recoverable one: the operator must remain compatible with the Kubernetes API, its CRDs, and any objects it has already created.
Consider adopting a new secrets service. Its operator might watch a custom resource, create Secrets, rotate credentials, and update workload references. If that controller stops reconciling, existing Pods may continue serving traffic while newly scheduled Pods fail to start with fresh credentials. That failure can evade a service availability alert because the current replicas still answer requests. The on-call engineer then needs to determine whether the problem is the external service, the controller, RBAC, a webhook, or an incompatible CRD version. The “automatic” path has created a production debugging path that did not previously exist.
The Maintenance Tax Your Adoption Plan Is Missing is a useful warning about recurring work, but an operator adds a sharper question for DevOps: does the new tool need its own control plane inside this cluster? If the answer is no, do not grant it cluster-wide permissions merely because the vendor’s quick-start uses them. Permissions and controllers persist after the adoption project closes, so their cost falls on whoever handles the next incident and Kubernetes upgrade.
The installation checklist should therefore begin with failure behavior, not feature count. Ask what happens when the operator Deployment has zero ready replicas, when its certificate expires, and when a CRD conversion fails during an upgrade. Ask whether deleting the Helm release leaves custom resources or finalizers behind. Those questions matter because Kubernetes will not treat an application’s custom resources as disposable just because the team has stopped using its chart.
A plain Deployment wins until reconciliation solves a demonstrated problem
There are two real options. A plain Deployment managed by Helm 3 or Argo CD wins when the tool runs as a service or agent and its configuration changes through a controlled release. It costs the team explicit upgrade steps and, where needed, a small amount of automation around configuration. A Kubernetes operator wins when the tool must continually react to cluster events, manage many dependent resources, or recover safely from drift that a release pipeline cannot handle. It costs controller upgrades, CRD compatibility testing, RBAC review, and another component to diagnose during an incident.
For an OpenTelemetry Collector gateway, a Deployment with a ConfigMap is often the better first installation: its job is to receive and export telemetry, not to reconcile application resources. A collector that needs per-node access can run as a DaemonSet without requiring an operator. Conversely, cert-manager earns its controller model because certificates expire and need ongoing reconciliation against issuers and Kubernetes Secrets. The distinction is the work the controller must perform between releases, not whether the vendor offers an operator chart.
I would not install a vendor operator across every cluster during the first adoption sprint, because a fleet-wide rollout makes one untested upgrade path a fleet-wide incident path. Begin in one noncritical cluster with the smallest RBAC scope the product supports. Before widening that scope, demonstrate a task that fails or becomes unreasonably manual without reconciliation. “The operator makes installation easier” is insufficient justification, because Helm and Argo CD can already repeat an installation without adding a permanently running controller.
Use a proposed 30-day trial cap—a planning value to adjust to your release cadence—to force a decision rather than leaving a trial controller installed indefinitely. At the end, either record the reconciliation work it performed and approve its support model, or remove it and check for remaining CRDs, webhooks, and finalizers. The removal check matters because these resources can outlive the workload and affect later deployments.
A successful install proves less than a failed-controller rehearsal
Before approving an operator, inventory what already participates in the Kubernetes API path. The following commands run with cluster-wide read permissions and expose CRDs, admission webhooks, aggregated APIs, and the workloads you will need to inspect. They do not prove that a candidate operator is safe; they establish what it might have to coexist with.
#!/usr/bin/env bash set -euo pipefail kubectl config current-context kubectl get crds -o name kubectl get mutatingwebhookconfigurations -o name kubectl get validatingwebhookconfigurations -o name kubectl get apiservices -o name kubectl get deployments -A kubectl get pods -A --field-selector=status.phase!=Running
Run the inventory before and after installation and review the difference, particularly any webhook with a broad namespace selector. A validating webhook can block writes when it is unavailable if its failurePolicy is set to Fail; a policy that fails open may preserve writes but stop enforcing a rule. Neither setting is universally correct, because availability and enforcement serve different purposes. The production requirement should decide the setting, and the failure test should verify the outcome.
Then stop the controller in the test cluster. Scale its Deployment to zero, create or update a representative custom resource, and observe what happens to both the existing workload and a newly deployed one. Restore the controller and time how long it takes to reconcile the pending change. Repeat with a rejected manifest and with an upgrade that changes the CRD schema. These tests are more revealing than a green install because they exercise the conditions an on-call engineer must resolve without the implementation team present.
Set an initial 15-minute recovery target as a value to tune against your incident response process: can the responder restore reconciliation or roll back the release within that window? Record the elapsed time you actually observe rather than marking the test “passed” because the controller eventually recovered. If recovery requires editing an undocumented custom resource or deleting a finalizer by hand, the trial has found an operational dependency, not a documentation footnote.
The support contract should be written before the custom resource
An operator’s support contract should name an owner for the controller image, the Helm chart, and every CRD version in use. It should also name the team that approves Kubernetes upgrades. This division matters because an application team can update its custom resources while the platform team remains responsible for an incompatible controller after a cluster upgrade. Put the supported version combinations in the repository beside the deployment manifests, where the person preparing an upgrade can find them.
The Maintenance Tax PMs Miss When Adopting New Tech is a useful warning to product managers, but the production acceptance test belongs to the team carrying the pager. Require an executable rollback procedure, not a promise that the vendor supports rollback. CRD storage-version changes and conversion webhooks can make “helm rollback” insufficient, because restoring an older controller does not necessarily restore objects to an older schema. Test rollback with representative custom resources before relying on it in a maintenance window.
Monitor the boundary the operator can break. Prometheus documents 1m as its default scrape_interval, but that vendor-published default is not an incident policy: choose scrape and alert intervals based on how quickly failed reconciliation harms workloads. Watch controller availability and restarts, reconcile errors, queue depth if the controller exports it, and Kubernetes API latency through metrics such as apiserver_request_duration_seconds. If admission webhooks are installed, examine apiserver_admission_webhook_admission_duration_seconds as well. Alertmanager should route these alerts to an owner who can act, rather than sending every controller warning to a shared channel.
A sample service objective of 99.9% availability is an illustrative target, not evidence that the operator is reliable. Define what users experience when reconciliation stops: perhaps existing traffic works, but credential rotation fails or new replicas cannot become ready. Those symptoms need separate checks because a successful HTTP probe against the current replicas cannot detect a failed future rollout. Tie the adoption decision to those checks and to a rehearsed rollback; otherwise the team is accepting a dependency it cannot yet measure or remove.
Make the next adoption request prove its control-plane need
At the next tool review, ask its sponsor for one concrete operation that requires continuous in-cluster reconciliation. If a Deployment, DaemonSet, or scheduled job can do the work, trial that path first. If an operator is necessary, book the failed-controller and rollback rehearsals before production installation, and assign the pager owner now. That small gate prevents installation convenience from silently becoming a permanent cluster support obligation.
