provider-kubernetes

provider-kubernetes is a Crossplane Provider that enables deployment and management of arbitrary Kubernetes objects on clusters typically provisioned by Crossplane:

• A Provider resource type that only points to a credentials Secret.
• An Object resource type that is to manage Kubernetes Objects.
• A managed resource controller that reconciles Object typed resources and manages arbitrary Kubernetes Objects.

Install

If you would like to install provider-kubernetes without modifications, you may do so using the Crossplane CLI in a Kubernetes cluster where Crossplane is installed:

crossplane xpkg install provider xpkg.crossplane.io/crossplane-contrib/provider-kubernetes:v1.0.0

You may also manually install provider-kubernetes by creating a Provider directly:

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-kubernetes
spec:
  package: xpkg.crossplane.io/crossplane-contrib/provider-kubernetes:v1.0.0

Target cluster clients

The provider builds one client per distinct ProviderConfig credential set (kubeconfig plus identity) and keeps it in an LRU cache. Building a client is expensive because its REST mapper primes itself with full aggregated discovery on first use, so the cache has to hold every credential set the provider talks to. Once the credential sets in use outnumber the bound, every new client evicts the least recently used one, whose next reconcile rebuilds it (discovery included) and evicts another: the whole cache churns, not only the credential set that did not fit.

At startup the provider sizes the cache from the cluster: it lists its ProviderConfig and ClusterProviderConfig objects, derives the cache key of each one the way the client builder does (so ProviderConfigs sharing a kubeconfig count once, and every in-cluster ProviderConfig counts as one), and bounds the cache at the number of distinct credential sets plus headroom of ten percent or four entries, whichever is larger, for credentials that rotate and ProviderConfigs created later, never below 8. The result is logged at startup. The provider does not start if its ProviderConfigs and their credentials cannot be read within 300 seconds. When the provider runs outside a cluster, as in local development, ProviderConfigs with an in-cluster identity cannot be resolved and each counts as a credential set of its own, so the cache is merely oversized.

The bound is fixed for the lifetime of the process, so ProviderConfigs created after startup share the headroom until the next restart. The cache reports provider_kubernetes_client_cache_size (the bound), provider_kubernetes_client_cache_entries (clients currently cached) and provider_kubernetes_client_cache_events_total with an event label of hit, miss or evict; a sustained eviction rate means the credential sets in use no longer fit the bound and the cache is churning.

Client-side rate limiting is disabled for these clients. A cached client is shared by every concurrent reconcile of its credential set, so client-go's per-client token bucket (5 QPS, burst 10) would serialize them. This applies to everything built from the same configuration: the reconcile client, the watch informers and the server-side apply discovery client. Load on the target cluster is bounded by --max-reconcile-rate on the provider side and by API Priority and Fairness on the API server.

Known limitations

Write-only fields cannot be reconciled

Some Kubernetes fields are "write-only": the API server accepts them but transforms or drops them on write, so they never appear when the resource is read back. The canonical example is Secret.stringData, which the API server merges into data without ever storing stringData itself.

The provider cannot correctly reconcile such fields in an Object manifest (see the analysis in #420):

• With server-side apply (the default since v1.2.0), drift detection compares the fields owned by the provider's field manager on the observed resource against the result of a dry-run apply of the manifest. A write-only field appears in neither, so changing e.g. stringData in the manifest after creation produces no diff and is never applied to the cluster. The provider has no way of knowing that data is the server-side transformation of stringData.
• If the provider's field manager owns only fields that are absent from the observed resource — a Secret whose manifest sets only stringData (#420), or one created with an empty data (#418) — the managed-fields extraction fails with ExtractInto ... expected map, got <nil> and the Object gets stuck with Synced: False in ReconcileError.

Prefer round-trippable fields in Object manifests: for Secrets, set base64-encoded values in data instead of using stringData.

Developing locally

See the header of go.mod for the minimum supported version of Go.

Start a local development environment with Kind where crossplane is installed:

make
make local-dev

Now you can either run the controller locally or in-cluster.

Running locally

Run controller locally against the cluster:

make run

Since the controller is running outside the Kind cluster, you need to make the API server accessible to the controller. You can do this by running a proxy:

# on a separate terminal
sudo kubectl proxy --port=8081

See below for how to properly setup the RBAC for the locally running controller.

Running in-cluster

Run controller in-cluster:

make local-deploy

See below for how to properly setup the RBAC for the locally running controller.

Required configuration

1. Prepare provider config for the local cluster:

2. If provider kubernetes running in the cluster (e.g. provider installed with crossplane or using make local-deploy):

   SA=$(kubectl -n crossplane-system get sa -o name | grep provider-kubernetes | sed -e 's|serviceaccount\/|crossplane-system:|g')
   kubectl create clusterrolebinding provider-kubernetes-admin-binding --clusterrole cluster-admin --serviceaccount="${SA}"
   kubectl apply -f examples/namespaced/provider/config-in-cluster.yaml
   

3. If provider kubernetes running outside the cluster (e.g. running locally with make run)

   KUBECONFIG=$(kind get kubeconfig --name local-dev | sed -e 's|server:\s*.*$|server: http://localhost:8081|g')
   kubectl -n crossplane-system create secret generic cluster-config --from-literal=kubeconfig="${KUBECONFIG}"
   kubectl apply -f examples/namespaced/provider/config.yaml
   

4. Now you can create Object resources with provider reference, see sample object.yaml.

   kubectl create -f examples/namespaced/object/object.yaml
   

Cleanup

To delete the local kind cluster:

make controlplane.down