Skip to content

BigTable Autoscaling - HTTP 499 error #673

Description

@tommycouri-bestbuy

Checklist

Bug Description

When trying to deploy BigTable with autoscaling through config connector, I'm seeing the following error:

Cancelled (HTTP 499): Operation successfully rolled back : Both manual scaling (serve_nodes) and autoscaling (cluster_autoscaling_config) enabled. Exactly one must be set for CreateInstance/CreateCluster

Because of this, the BigTable instance does not deploy at all. When I try and deploy an instance without autoscaling enabled, it deploys without any issue. I have also opened GCP support case 29934946 regarding this issue. Their recommendation was to open up a GitHub issue here, as they believe the issue sits within Config Connector.

I have tried deploying this with and without the numNodes value as well. When that is not present, the instance fails to deploy without any error message in the activity log. I am also running config connector V1.88

Additional Diagnostic Information

Here is my sample yaml file that I'm trying to deploy:

  apiVersion: bigtable.cnrm.cloud.google.com/v1beta1
  kind: BigtableInstance
  metadata:
    name: bigtableinstance-auto
  spec:
    displayName: BigtableAuto
    cluster:
      - autoscalingConfig:
          cpuTarget: 60
          maxNodes: 3
          minNodes: 1
        numNodes: 1
        clusterId: bigtable-auto
        zone: us-central1-a

Kubernetes Cluster Version

1.21.0

Config Connector Version

1.88

Config Connector Mode

namespaced mode (default)

Log Output

Output from kubectl describe. I have removed my Namespace information:

API Version:  bigtable.cnrm.cloud.google.com/v1beta1
Kind:         BigtableInstance
Metadata:
  Creation Timestamp:  2022-06-23T19:15:55Z
  Finalizers:
    cnrm.cloud.google.com/finalizer
    cnrm.cloud.google.com/deletion-defender
  Generation:  1
  Managed Fields:
    API Version:  bigtable.cnrm.cloud.google.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:finalizers:
          .:
          v:"cnrm.cloud.google.com/deletion-defender":
          v:"cnrm.cloud.google.com/finalizer":
      f:status:
        .:
        f:conditions:
        f:observedGeneration:
    Manager:      cnrm-controller-manager
    Operation:    Update
    Time:         2022-06-23T19:15:55Z
    API Version:  bigtable.cnrm.cloud.google.com/v1beta1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:kubectl.kubernetes.io/last-applied-configuration:
      f:spec:
        .:
        f:cluster:
        f:displayName:
    Manager:         kubectl-client-side-apply
    Operation:       Update
    Time:            2022-06-23T19:15:55Z
  Resource Version:  220808017
  UID:               ef40cdb9-da1e-4333-ae84-cbbdaa4e82c8
Spec:
  Cluster:
    Autoscaling Config:
      Cpu Target:  60
      Max Nodes:   3
      Min Nodes:   1
    Cluster Id:    bigtable-auto
    Num Nodes:     1
    Zone:          us-central1-a
  Display Name:    BigtableAuto
Status:
  Conditions:
    Last Transition Time:  2022-06-23T19:15:55Z
    Message:               Update call failed: error applying desired state: summary: Error creating instance. rpc error: code = Canceled desc = Operation successfully rolled back : Both manual scaling (serve_nodes) and autoscaling (cluster_autoscaling_config) enabled. Exactly one must be set for CreateInstance/CreateCluster
    Reason:                UpdateFailed
    Status:                False
    Type:                  Ready
  Observed Generation:     1
Events:
  Type    Reason    Age                    From                         Message
  ----    ------    ----                   ----                         -------
  Normal  Updating  5m27s (x570 over 19h)  bigtableinstance-controller  Update in progress

Steps to reproduce the issue

Deploying the following yaml file will should reproduce the issue.

YAML snippets

apiVersion: bigtable.cnrm.cloud.google.com/v1beta1
kind: BigtableInstance
metadata:
  name: bigtableinstance-auto
spec:
  displayName: BigtableAuto
  cluster:
    - autoscalingConfig:
        cpuTarget: 60
        maxNodes: 3
        minNodes: 1
      numNodes: 1
      clusterId: bigtable-auto
      zone: us-central1-a

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions