cnrm-resource-stats-recorder crash loop triggered by recorder container; port 8888 already in use #449

Checklist

• I did not find a related open issue.
• I did not find a solution in the troubleshooting guide: (https://cloud.google.com/config-connector/docs/troubleshooting)
• If this issue is time-sensitive, I have submitted a corresponding issue with GCP support.

Bug Description

A new revision for cnrm-resource-stats-recorder that tried to start yesterday is failing in a crash loop in one of my clusters, complaining that port 8888 is already in use (in the 'recorder' container).

It's trying to run recorder: gcr.io/gke-release/cnrm/recorder:d399cc9, prom-to-sd: k8s.gcr.io/prometheus-to-sd:v0.9.1.
The previous version is still running fine, on versions: recorder: gcr.io/gke-release/cnrm/recorder:2081072, prom-to-sd: k8s.gcr.io/prometheus-to-sd:v0.9.1.

Any insight into this issue would be appreciated!
Should I kill this revision and try re-applying the config connector manifests?

Additional Diagnostic Information

Kubernetes Cluster Version

1.17.17-gke.2800

Config Connector Version

1.39.0

Config Connector Mode

cluster

Log Output

The logs from the "recorder" container are (repeated with each crash):

{ "msg": "Recording the stats of Config Connector resources" }
{ "error": "listen tcp :8888: bind: address already in use", "msg": "error registering the Prometheus HTTP handler" }

Steps to Reproduce

Steps to reproduce the issue

I'm not sure exactly what triggered this issue, but it seems that it occurred when the recorder container was trying to upgrade to a new version while another revision was already running in the cluster.