Spots

Manually Triggering Reconciliation in My Kubernetes Operator

In the previous post, I explored what happens when my Kubernetes operator fails. I created a real failure by removing ConfigMap creation permission from the operator. The controller failed, retried several times, and eventually stopped retrying. Then I restored the RBAC permissions. But something unexpected happened. The operator now had permission to create the ConfigMap again, but the failed Greeting was not immediately reconciled. That led me to another question: How can I manually ask the controller to reconcile a resource again without changing its actual specification? How can I manually ask the controller to reconcile a resource again without changing its actual specification? That is what I wanted to solve next. The failure flow looked like this: Then I fixed the external problem: But fixing the ClusterRole did not produce a new event for the Greeting controller.

So the resource remained in its failed state

So the resource remained in its failed state. Previously I recovered by changing: That worked because changing the spec caused another reconciliation. But changing business configuration only to make the controller run again did not feel right. I wanted something more explicit. I decided to use a Kubernetes annotation: the annotation gets a new timestamp. Every time I want another reconciliation, I update this value. This means I can request another reconciliation without changing: There was one more problem. The Java Operator SDK normally avoids triggering reconciliation for every metadata-only update. should not necessarily cause the controller to run again. For normal desired-state changes, Kubernetes updates: So the controller can react when the actual spec changes. But my manual reconciliation only changes an annotation. So I needed a custom update filter.

The filter looks roughly like this: This keeps

The filter looks roughly like this: This keeps the controller focused on meaningful updates. One option would have been to simply accept every resource update. But that would be too broad. The operator itself updates: and status updates also change the Kubernetes resource. If every update triggered another reconciliation, it could create unnecessary reconciliation cycles. The custom filter lets me control that. I then configured the GreetingReconciler to use the custom update filter. The controller already had the managed ConfigMap workflow, so the overall structure becomes: Since I am already using Task as the developer interface for the project, I added: The task updates the annotation with the current timestamp. This makes manual reconciliation much easier to use during development. I first tested it on the healthy hello Greeting.

Then checked the annotation: The annotation contained the

Then checked the annotation: The annotation contained the new timestamp. Then I checked the Greeting Events: The interesting part was: Kubernetes had aggregated two equivalent Reconciled Events. One came from the normal reconciliation. The second came from my manual reconciliation request. So the annotation was doing exactly what I wanted. Something else was interesting. After manual reconciliation, I still had: At first this might look strange. But it is actually correct. The manual reconciliation changed: So Kubernetes did not increment: Before manual reconciliation: After manual reconciliation: The operator simply reconciled generation 1 again. This gave me a useful distinction. The healthy-resource test proved that the annotation worked. But the real reason I added this feature was failure recovery. So I repeated the RBAC failure experiment.

First, I edited the operator's ClusterRole: The normal

First, I edited the operator's ClusterRole: The normal ConfigMap permissions looked like: For the experiment I temporarily changed them to: I created another Greeting directly: The operator attempted to create: The Greeting eventually moved to: and automatic retries started. After the retry limit was reached, it remained failed. I restored the repository RBAC configuration: Previously, at this point I had changed: to trigger another reconciliation. This time I did not touch the specification. The update filter accepted the update. The controller reconciled the same desired state again. And this time ConfigMap creation succeeded. The new recovery flow became: This feels much cleaner than modifying application configuration only to wake the controller up. At this point my controller can be reconciled for several different reasons. All of them eventually call the same controller logic.

That also reinforces another important idea: Reconciliation should

That also reinforces another important idea: Reconciliation should be safe to run multiple times. Reconciliation should be safe to run multiple times. The controller should keep trying to move actual state toward desired state rather than assuming it runs only once. The operator now looks roughly like: The complete implementation is available in my Platform Lab repository. The changes covered in this post are available in: The main changes include: reconcile-at annotation support manual reconciliation Taskfile command filtering spec changes and manual reconciliation separately The main thing I learned from this step is that: are two different operations. Manual reconciliation means: That is why keeping the manual trigger in metadata rather than the resource specification feels like a better fit.

It also helped me understand why: can remain

It also helped me understand why: can remain unchanged even though the controller ran multiple times. The generation represents the desired configuration, not the number of reconciliation attempts. There is still more I want to understand around reconciliation itself. Some areas I want to explore are: After that, I want to move into another important part of the operator lifecycle:

For now, the Greeting operator supports both automatic

For now, the Greeting operator supports both automatic reconciliation and an explicit way to ask it to evaluate the current desired state again. For further actions, you may consider blocking this person and/or reporting abuse

News

Manually Triggering Reconciliation in My Kubernetes Operator

In the previous post, I explored what happens when my Kubernetes operator fails.

@spots #dev
Source: Dev.to
See more like this