Skip to content

fix(bigtable): sessionTableHandle self-heals across cache eviction - #20296

Merged
sushanb merged 4 commits into
googleapis:mainfrom
sushanb:fix/bigtable-session-table-cache-evict-self-heal
Aug 3, 2026
Merged

fix(bigtable): sessionTableHandle self-heals across cache eviction#20296
sushanb merged 4 commits into
googleapis:mainfrom
sushanb:fix/bigtable-session-table-cache-evict-self-heal

Conversation

@sushanb

@sushanb sushanb commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Summary

`TableShim` caches a `*sessionTableHandle` at Open time and holds it for the process lifetime. When `sessionTableCache`'s TTL sweeper evicts that handle (default 1h idle), subsequent `ReadRow` / `Apply` through the stale pointer hit a `Close`d pool and return wrapped `btransport.ErrPoolClosed`. That sentinel is `codes.Unknown` (not `codes.Unimplemented`), so `InterceptUnimplemented` does NOT fall back to classic. The user's `*Table` stays poisoned until process restart.

Fix (Design C)

Extract an iterative `dispatch()` helper on `*sessionTableHandle`:

  • `Close()` sets `evicted atomic.Bool` BEFORE `api.Close()` so future readers observe eviction before the underlying pool tears down.
  • `ReadRow` / `MutateRow` call `dispatch()`, which atomic-Loads `evicted` on the happy path (one atomic per RPC) and, on eviction, routes through `resolveSuccessor` — identity-checked `removeEntry` + `getOrOpen` — to install or find a live successor and dispatch there. Loops (not recursion) to survive a tight-TTL test where the fresh successor itself gets evicted mid-flight.
  • Cache is closed → `resolveSuccessor` returns nil → `dispatch` falls through to the original doomed handle → the surviving `h.api` surfaces the honest terminal error rather than looping.

Design tradeoffs considered

  • Per-RPC provider closure: ~30 ns / RPC cache-map lookup on every call. Rejected — hot-path cost.
  • Cache-side ErrPoolClosed catch: leaks the transport-layer error sentinel into the cache/handle layer. Rejected — layering.
  • Design C (this PR): ~2 ns / RPC atomic Load on the happy path; ~30 ns Open-time cost once per table opened.

Test plan

  • `TestSessionTableHandle_EvictedSelfHeals` — post-`Close`, ReadRow / MutateRow succeed via a freshly minted successor. `openFn` re-invoked; cache holds the successor with distinct identity.
  • `TestSessionTableHandle_SweeperEvictionSelfHeals` — the actual production trigger. Advances a fake clock past TTL, drives `sweepOnce`, asserts the held pointer still services RPCs.
  • `TestSessionTableHandle_EvictedReopenFailsGracefully` — cache closed → ReadRow returns within a bounded 1 s deadline (no loop) and cache stays empty (no leaked successor).
  • `TestSessionTableHandle_ConcurrentEvictAndReadRace` — `-race`: one goroutine spins `Close`+`getOrOpen` cycles while another spins `ReadRow` through the original held handle. Clean.
  • `TestSessionTableHandle_EvictionStormConvergesOnSingleInstalled` — N=32 concurrent post-eviction RPCs converge on exactly one installed successor; every loser api is `Close`d.
  • Existing `TestSessionTableCache_*` suite unchanged and passing.
  • `go test ./ -race -count=1 -short` clean.

Bug reference: internal audit finding #6 ("TTL evict → ErrPoolClosed").

sushanb added 2 commits August 3, 2026 03:09
#6)

TableShim caches a *sessionTableHandle at Open time and holds it for
the process lifetime. When the sessionTableCache TTL sweeper evicts
that handle, subsequent ReadRow/Apply through the stale pointer hit a
Close'd pool and return wrapped btransport.ErrPoolClosed. Because that
sentinel is codes.Unknown (not codes.Unimplemented), InterceptUnimplemented
does not fall back to classic. The user's *Table stays poisoned until
process restart. Reproduced end-to-end against sandbox in
session_table_evict_repro_test.go (pre-fix FAIL / post-fix PASS).

Fix (Design C): keep TableShim caching the resolved handle at Open
time so the RPC hot path pays only an atomic Load. Move all recovery
logic INTO the handle:

  - sessionTableHandle gains an evicted atomic.Bool + captured openFn.
  - Close() sets evicted.Store(true) BEFORE api.Close so concurrent
    readers see the flag and route through the self-heal path.
  - ReadRow / MutateRow atomic-Load evicted; on true, call
    reopenAfterEviction (identity-checked removeEntry + getOrOpen)
    to get a live successor and dispatch on it. Happy path is
    unchanged: atomic Load + touch + api.ReadRow.

Trade-offs vs alternative designs considered on this branch:
  - Per-RPC provider closure (03a8dcde89, reverted): ~30ns/RPC cache-
    map lookup on every call. This design: ~2ns/RPC atomic Load.
    Cost moves back to Open time (~30ns once per table opened).
  - Cache-side ErrPoolClosed catch: leaked transport-layer error
    sentinel into the cache/handle layer.

Nothing changes in TableShim, open.go, or the test files that access
shim.session. The fix is entirely inside session_table_cache.go —
correct layer for eviction semantics.

Change surface:
  - bigtable/session_table_cache.go   +openFn, +evicted atomic.Bool,
                                       +reopenAfterEviction, wrap
                                       ReadRow/MutateRow, update Close
                                       ordering.
  - bigtable/open_test.go             +TestSessionTableHandle_EvictedSelfHeals
                                       (pins openFn re-invocation +
                                       cache re-install + handle-
                                       identity stability).
                                       +TestSessionTableHandle_EvictedReopenFailsGracefully
                                       (pins terminal cache-close
                                       branch doesn't loop).
  - bigtable/docs/specs/SESSION_POOL_SPEC.md  #3 TableShim bullet
                                              extended: handle self-
                                              heal semantics + link
                                              to session_table_cache.go
                                              invariants + test name.
Addresses the round-2 reviews on 82cd0a3 across three reviewers.

Iterative dispatch() helper (thermo + igor blockers). ReadRow /
MutateRow are shape-twins; both used to duplicate the self-heal
conditional (drifting at comment level already, would drift at logic
level as future methods land). Extract dispatch() on *sessionTableHandle
that returns the live handle via a for-loop (not recursion — igor's
catch: a hair-trigger TTL test could otherwise recurse if the fresh
successor gets evicted mid-flight). ReadRow / MutateRow collapse to
3-line pass-throughs: dispatch → touch → api.op. Any future proxied
method drops in as a 3-liner with zero branch structure to keep in
sync. Bonus: touch fires on the live handle, not the doomed original.

Rename reopenAfterEviction → resolveSuccessor (igor). "Reopen" reads
as "revive h in place"; we're actually asking the cache for whoever's
live under h.key now — could be a peer's winner, could be a fresh
one we just minted. New name is honest about what happens.

Kill dead fresh==h guard (sushanb + thermo). After removeEntry(h),
getOrOpen can't return h (hit branch requires entries[key]==h, slow
path mints a fresh *sessionTableHandle). The !ok half of the type
assertion stays as a defensive check but the fresh==h half is
provably unreachable.

Close-ordering doc reworded (all three reviewers). "Nanoseconds"
window claim was wrong — a reader preempted between Load(false) and
api.dispatch can span arbitrary wall-clock under scheduler pressure.
Rewrite to the honest three-step invariant: (1) evicted.Store(true)
routes future readers through self-heal, (2) removeEntry drops from
the map, (3) api.Close tears down. (1)→(3) is the load-bearing
ordering; (2) tightens the window. Also documents which readers the
flag actually helps (post-Load-true only, not in-flight pre-Load).

WHY on openFn field (sushanb + igor). Documents the set-once
*Client.sessionImpl invariant (if that ever becomes reconfigurable,
self-heal will stale) AND the "invoked only via getOrOpen which
nil-guards on close" invariant (direct openFn calls would lose the
guard).

Fix EvictedReopenFailsGracefully (all three). Was Logf-not-assert;
docstring claimed to test a deadApi that wasn't wired. Rewritten:
wrap ReadRow in context.WithTimeout(1s) for hang detection; assert
cache.entries stays empty after the call (the closed-cache guard
inside getOrOpen must have refused to install the freshly-opened
api, releasing it — otherwise we'd leak).

Three new tests (sushanb):
 - TestSessionTableHandle_SweeperEvictionSelfHeals — the actual
   production trigger. Prior tests used explicit handle.Close; this
   drives sweepOnce after TTL and asserts the held pointer still
   works.
 - TestSessionTableHandle_ConcurrentEvictAndReadRace — -race stress:
   evictor spins Close+reopen while reader spins ReadRow through the
   original held handle.
 - TestSessionTableHandle_EvictionStormConvergesOnSingleInstalled —
   N=32 concurrent post-eviction RPCs. Pins the real invariant
   getOrOpen guarantees (exactly one installed successor + every
   loser api closed), not the aspirational single-flight-openFn one
   my first draft assumed. Making openFn genuinely single-flight
   would be a real perf win but is out of scope for this fix.

Split test file (thermo blocker). open_test.go crossed 1006 lines
after the new tests landed. All cache-focused tests + helpers +
fakes (fakeClock, newTestSessionTableCache, openNoop, openClosing,
closeCountingTable, all TestSessionTableCache_* and
TestSessionTableHandle_*) move to a new session_table_cache_test.go
(597 lines). open_test.go drops to 629. Both under threshold.

SPEC trim (sushanb + igor). Trimmed the TableShim bullet from a
paragraph to an invariant + Java-parity note ("Go-specific hazard
— Java's session-table cache has no TTL sweeper, so Design C has
no direct Java analog").

Non-blockers deferred:
 - Single-flight openFn in getOrOpen (would save N-1 pool creations
   in an eviction storm; separate perf follow-up).
 - Narrowing getOrOpen's return to *sessionTableHandle (thermo nit;
   touches the interface at the public boundary — separate PR).
@sushanb
sushanb requested review from a team as code owners August 3, 2026 15:01
@product-auto-label product-auto-label Bot added the api: bigtable Issues related to the Bigtable API. label Aug 3, 2026

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request implements a self-healing mechanism for sessionTableHandle in the Bigtable session pool, allowing evicted handles to transparently resolve and route subsequent RPCs (such as ReadRow and MutateRow) through a freshly-minted successor handle. Unit tests for the cache and handle have been moved to a dedicated test file, with new tests added to verify self-healing, concurrent races, and eviction storms. The review feedback recommends optimizing the self-healing path to avoid acquiring the global cache mutex repeatedly after eviction by caching the resolved successor handle directly inside the evicted handle using an atomic.Pointer and applying path compression in dispatch().

Comment thread bigtable/session_table_cache.go Outdated
Comment on lines 86 to 89
evicted atomic.Bool
closeOnce sync.Once
closeErr error
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

To avoid a performance bottleneck where every single RPC after the first eviction has to acquire the global cache mutex twice (via removeEntry and getOrOpen), we can cache the resolved successor directly in the evicted handle using an atomic.Pointer[sessionTableHandle].

Suggested change
evicted atomic.Bool
closeOnce sync.Once
closeErr error
}
evicted atomic.Bool
sucessor atomic.Pointer[sessionTableHandle]
closeOnce sync.Once
closeErr error
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch — adopted in 2552f50. Under TableShim's process-lifetime pointer-caching pattern, every RPC on a stale handle would pay 2× cache.mu forever without this. Field is now atomic.Pointer[sessionTableHandle] (fixed typo sucessorsuccessor) with a WHY comment on the field explaining why the Store races are safe by construction (getOrOpen's at-most-one-per-key invariant makes concurrent Stores idempotent; a divergent-next race just walks one extra hop on the next dispatch — no correctness bug, just lagging compression).

Comment on lines +108 to +118
func (h *sessionTableHandle) dispatch() *sessionTableHandle {
cur := h
for cur.evicted.Load() {
next := cur.resolveSuccessor()
if next == nil {
return cur // reopen refused; let cur.api surface the terminal error
}
cur = next
}
return cur
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

Update dispatch() to use the cached successor pointer. If a successor is already resolved, we can load it atomically without acquiring any locks. We also apply path compression when returning the active successor so that intermediate evicted handles can be garbage collected and subsequent lookups are O(1) atomic loads.

func (h *sessionTableHandle) dispatch() *sessionTableHandle {
	cur := h
	for cur.evicted.Load() {
		if next := cur.sucessor.Load(); next != nil {
			cur = next
			continue
		}
		next := cur.resolveSuccessor()
		if next == nil {
			return cur // reopen refused; let cur.api surface the terminal error
		}
		cur.sucessor.Store(next)
		cur = next
	}
	if cur != h {
		h.sucessor.Store(cur) // path compression to bypass intermediate evicted handles
	}
	return cur
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopted in 2552f50 with two small modifications:

  1. Single Store(next) site inside the loop rather than two — the per-iteration Store handles the fresh-resolve case; the terminal h.successor.Store(cur) handles the "walked a chain" case. Same shape, one Store to reason about.
  2. Preserved the existing dispatch() doc block (eviction branch + loop rationale + cache-closed fallthrough) and added the path-compression paragraph on top of it.

Final shape:

func (h *sessionTableHandle) dispatch() *sessionTableHandle {
	cur := h
	for cur.evicted.Load() {
		next := cur.successor.Load()
		if next == nil {
			next = cur.resolveSuccessor()
			if next == nil {
				return cur // reopen refused; let cur.api surface the terminal error
			}
			cur.successor.Store(next)
		}
		cur = next
	}
	if cur != h {
		h.successor.Store(cur)
	}
	return cur
}

Coverage: TestSessionTableHandle_EvictionStormConvergesOnSingleInstalled already exercises N=32 concurrent post-eviction dispatch calls under -race, which is where the successor-cache races would surface. No new test needed.

sushanb added 2 commits August 3, 2026 15:09
Follow-up to igor round-2 on PR googleapis#20296.

 - Typo: openFn doc "staless" → "goes stale".
 - openFn field: add invariant note that resolveSuccessor threads the
   captured closure through unchanged, so all handles that ever share
   the key share the same openFn (guards future contributors from
   silently diverging successors by threading a per-call openFn
   override into getOrOpen).
 - SPEC bullet: clarify TableShim's `session` field is typed
   session.TableAPI (concrete value happens to be *sessionTableHandle);
   drop Java-parity phrasing per author preference.
 - Trim double blank line above the "─── sessionTableCache tests" divider.

No behavior change.
Adopts gemini-code-assist's PR googleapis#20296 review suggestion (with
modifications approved by igor round-2):

Add `successor atomic.Pointer[sessionTableHandle]` to sessionTableHandle
and use it in dispatch() to cache the resolved live successor. Without
this, EVERY RPC on a stale handle pays 2× cache.mu (removeEntry +
getOrOpen) forever, because TableShim's cached pointer never gets
re-fetched. With it, only the FIRST post-eviction RPC pays that cost;
subsequent ones are one extra atomic.Load.

dispatch() also applies path compression on return so a chain of
evicted handles collapses on the next observation — union-find with
path compression, standard technique. Intermediate handles become
unreachable from h and are eligible for GC.

Store races are documented on the field and safe by construction:
  - Two concurrent dispatches on the same evicted handle race into
    getOrOpen, which is at-most-one-per-key (loser closes its local
    api and returns the winner's handle), so both racers see the
    same `next` — the Store is idempotent.
  - Divergent-`next` race (successor itself evicted between one
    dispatch's resolveSuccessor and another's Store) writes a stale
    pointer; the next dispatch's outer for loop walks one extra hop
    through it and updates. No correctness bug, just a lagging
    compression.

Preserves the existing dispatch() doc block (eviction branch + loop
rationale + cache-closed fallthrough) and adds the path-compression
paragraph.

Modifications from gemini's original suggestion:
  - Typo: sucessor → successor.
  - Single Store site inside the loop (gemini had two writes:
    per-iteration and at the return). One is enough — the
    per-iteration Store handles the fresh-resolve case; the terminal
    h.successor.Store(cur) covers the "we walked a chain" case.
  - Preserved the fallthrough-to-cur-on-nil semantic + comment.

No new tests — TestSessionTableHandle_EvictionStormConvergesOnSingleInstalled
already exercises N=32 concurrent post-eviction dispatch calls under
-race, which is where the successor-cache races would show up.
@sushanb
sushanb merged commit 0dd98cd into googleapis:main Aug 3, 2026
19 checks passed
sushanb pushed a commit that referenced this pull request Aug 3, 2026
🤖 I have created a release *beep* *boop*
---


##
[1.52.0](bigtable/v1.51.0...bigtable/v1.52.0)
(2026-08-03)


### Features

* **bigtable:** Add AFE picker (Simple / LeastInFlight / LeastLatency)
([#20204](#20204))
([bcbf714](bcbf714))
* **bigtable:** Add ClientConfig.DisableSession to opt out of session
backend
([#20297](#20297))
([7ee5e44](7ee5e44))
* **bigtable:** Add getClientConfigDirectAccessChecker for session pools
([#20209](#20209))
([3b8d30a](3b8d30a))
* **bigtable:** Add NoOpChannelPrimer for session channel pools
([#20208](#20208))
([d055a8a](d055a8a))
* **bigtable:** Add per-AFE sessionList for the two-tier session pool
([#20224](#20224))
([dbf0c3f](dbf0c3f))
* **bigtable:** Add protoRowToRow conversion helper for TableShim
([#20257](#20257))
([1297143](1297143))
* **bigtable:** Add Session debug surface (observability fields +
methods)
([#20211](#20211))
([d8d3e16](d8d3e16))
* **bigtable:** Add Session lifecycle (Start, Close, ForceClose,
readLoop, heartBeatLoop)
([#20215](#20215))
([b9e53c6](b9e53c6))
* **bigtable:** Add Session struct + state machine
([#20117](#20117))
([09acbb3](09acbb3))
* **bigtable:** Add session.Config.EnableDebug to gate sessionz debug
state
([#20247](#20247))
([ce74c31](ce74c31))
* **bigtable:** Add SessionClient + SessionTable + lazyPool
([#20228](#20228))
([ab2c96c](ab2c96c))
* **bigtable:** Add SessionPoolImpl (two-tier pool + scaling + debug)
([#20225](#20225))
([683eda8](683eda8))
* **bigtable:** Rename session pool display to
<resource-id>-<PERM>
([#20248](#20248))
([35e146e](35e146e))
* **bigtable:** Route Client.Open()-returned *Table through the Diverter
([#20273](#20273))
([2b81c7d](2b81c7d))
* **bigtable:** State-based classification for abnormal session close
([#20243](#20243))
([f2905b7](f2905b7))
* **bigtable:** TableShim fallback to classic on session UNIMPLEMENTED
([#20269](#20269))
([36540af](36540af))
* **bigtable:** TTL-on-idle cache for per-resource session.TableAPI
([#20263](#20263))
([00b2a49](00b2a49))
* **bigtable:** Wire Diverter on Client and route Open* via TableShim
([#20256](#20256))
([b32fbd7](b32fbd7))


### Bug Fixes

* **bigtable:** AFE picker latency signal — subtract poolWait and
compute TransportLatency = wire − backend at source
([#20281](#20281))
([bb8c4d5](bb8c4d5))
* **bigtable:** Guard NewStream OnFinish against grpc-go double-fire
([#20295](#20295))
([b51da29](b51da29))
* **bigtable:** Real per-resource pool teardown on sessionTable.Close +
cache close-race gate
([#20264](#20264))
([599aea9](599aea9))
* **bigtable:** Session.durations / session.uptime — set explicit
histogram bucket boundaries
([#20276](#20276))
([97eee22](97eee22))
* **bigtable:** SessionTableHandle self-heals across cache eviction
([#20296](#20296))
([0dd98cd](0dd98cd))
* **bigtable:** Translate ctx errors to gRPC status on session vRPC
([#20299](#20299))
([0f3b2a5](0f3b2a5))
* **bigtable:** Treat PingAndWarm NotFound as a successful prime
([#20219](#20219))
([a1557ad](a1557ad))


### Performance Improvements

* **bigtable:** Delete periodic Tick loop; sizing is event-driven
([#20285](#20285))
([2c096bd](2c096bd))
* **bigtable:** Drop pick_lost_race debug tag from CheckoutSession hot
path
([#20280](#20280))
([bd0e400](bd0e400))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

api: bigtable Issues related to the Bigtable API.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants