fix(bigtable): session.durations / session.uptime — set explicit histogram bucket boundaries - #20276
Conversation
…et boundaries
The OTel SDK default explicit-bucket boundaries cap at 10s
{0, 5, 10, 25, 50, ..., 5000, 7500, 10000}. Session lifetimes and
per-session uptimes are minutes-to-hours (an idle session lives
until server-driven scale-down, typically 4+ minutes). Every sample
past 10s lands in the last bucket, so p50/p95/p99 all render as
"10000ms" on dashboards — which is what you see today when
comparing the OTel session.durations metric against sessionz's
"Session lifetimes" (n=74, p50=4m21s, p95=4m50s, p99=4m54s):
the sessionz HDR histogram gets it right, the OTel metric flattens
everything to 10s.
Fix: register both histograms with explicit boundaries that match
java-bigtable's ClientSessionDuration / ClientSessionUptime
BUCKETS_MS — {0} + geometric doubling from 1ms to ~17.5min. Same
constant used for both, matching Java's shared BUCKETS_MS layout.
Introduces SessionLifetimeBounds next to the existing
FineGrainLatencyBounds so future metric additions have a clear
pick between "sub-ms latency" and "session-scale lifetime" bucket
sets.
session.open_latencies also gets explicit boundaries in this
change (FineGrainLatencyBounds) — same clip-at-10s bug technically
applies, but pre-open latency past 10s is rare so the visible
impact was smaller. Setting it here for consistency with the other
three session metrics and with transport_latencies.
No behaviour change for any other subsystem — bucket selection is
per-histogram at registration time.
There was a problem hiding this comment.
Code Review
This pull request configures explicit bucket boundaries for OpenTelemetry metrics in the Bigtable session tracer, applying custom boundaries to session durations, uptime, and open latencies. The review feedback suggests unexporting the newly introduced SessionLifetimeBounds variable to sessionLifetimeBounds to improve encapsulation, as it is only used internally within this package.
| // SessionLifetimeBounds matches java-bigtable's ClientSessionDuration | ||
| // / ClientSessionUptime BUCKETS_MS: {0} + geometric doubling from | ||
| // 1ms to ~17.5min. Sessions live minutes-to-hours; the OTel SDK | ||
| // default explicit-bucket boundaries cap at 10s, so p50/p95/p99 all | ||
| // clip to "10000ms" on the dashboard when session lifetimes exceed | ||
| // 10s. Shared by session.durations and session.uptime. | ||
| var SessionLifetimeBounds = func() []float64 { | ||
| b := []float64{0} | ||
| for v := float64(1); v <= 1_200_000; v *= 2 { | ||
| b = append(b, v) | ||
| } | ||
| return b | ||
| }() |
There was a problem hiding this comment.
Since SessionLifetimeBounds is only used within this package (and specifically within this file), it should be unexported (i.e., renamed to sessionLifetimeBounds) to prevent external packages from accessing or accidentally mutating this global slice. This improves encapsulation and avoids potential side effects.
| // SessionLifetimeBounds matches java-bigtable's ClientSessionDuration | |
| // / ClientSessionUptime BUCKETS_MS: {0} + geometric doubling from | |
| // 1ms to ~17.5min. Sessions live minutes-to-hours; the OTel SDK | |
| // default explicit-bucket boundaries cap at 10s, so p50/p95/p99 all | |
| // clip to "10000ms" on the dashboard when session lifetimes exceed | |
| // 10s. Shared by session.durations and session.uptime. | |
| var SessionLifetimeBounds = func() []float64 { | |
| b := []float64{0} | |
| for v := float64(1); v <= 1_200_000; v *= 2 { | |
| b = append(b, v) | |
| } | |
| return b | |
| }() | |
| // sessionLifetimeBounds matches java-bigtable's ClientSessionDuration | |
| // / ClientSessionUptime BUCKETS_MS: {0} + geometric doubling from | |
| // 1ms to ~17.5min. Sessions live minutes-to-hours; the OTel SDK | |
| // default explicit-bucket boundaries cap at 10s, so p50/p95/p99 all | |
| // clip to "10000ms" on the dashboard when session lifetimes exceed | |
| // 10s. Shared by session.durations and session.uptime. | |
| var sessionLifetimeBounds = func() []float64 { | |
| b := []float64{0} | |
| for v := float64(1); v <= 1_200_000; v *= 2 { | |
| b = append(b, v) | |
| } | |
| return b | |
| }() |
There was a problem hiding this comment.
Done in b895f044e6 — renamed to sessionLifetimeBounds. Confirmed grep of bigtable/ returns only session_tracer.go itself.
| "session.durations", | ||
| metric.WithDescription("Duration a session was alive (startTime → close)"), | ||
| metric.WithUnit("ms"), | ||
| metric.WithExplicitBucketBoundaries(SessionLifetimeBounds...), |
There was a problem hiding this comment.
Done in b895f044e6 — renamed to sessionLifetimeBounds. Confirmed grep of bigtable/ returns only session_tracer.go itself.
| "session.uptime", | ||
| metric.WithDescription("Age of currently-active sessions, sampled periodically"), | ||
| metric.WithUnit("ms"), | ||
| metric.WithExplicitBucketBoundaries(SessionLifetimeBounds...), |
There was a problem hiding this comment.
Done in b895f044e6 — renamed to sessionLifetimeBounds. Confirmed grep of bigtable/ returns only session_tracer.go itself.
Addresses gemini-code-assist review on PR googleapis#20276. The variable is only consumed within session_tracer.go (session.durations + session.uptime histogram registration). No external caller reaches for it, so keeping it exported broadens the internal/transport API surface without a reader. Mechanical rename SessionLifetimeBounds → sessionLifetimeBounds across all four occurrences in session_tracer.go. Build + vet clean; no other package touches the symbol.
|
Addressed in Note on the failing |
🤖 I have created a release *beep* *boop* --- ## [1.52.0](bigtable/v1.51.0...bigtable/v1.52.0) (2026-08-03) ### Features * **bigtable:** Add AFE picker (Simple / LeastInFlight / LeastLatency) ([#20204](#20204)) ([bcbf714](bcbf714)) * **bigtable:** Add ClientConfig.DisableSession to opt out of session backend ([#20297](#20297)) ([7ee5e44](7ee5e44)) * **bigtable:** Add getClientConfigDirectAccessChecker for session pools ([#20209](#20209)) ([3b8d30a](3b8d30a)) * **bigtable:** Add NoOpChannelPrimer for session channel pools ([#20208](#20208)) ([d055a8a](d055a8a)) * **bigtable:** Add per-AFE sessionList for the two-tier session pool ([#20224](#20224)) ([dbf0c3f](dbf0c3f)) * **bigtable:** Add protoRowToRow conversion helper for TableShim ([#20257](#20257)) ([1297143](1297143)) * **bigtable:** Add Session debug surface (observability fields + methods) ([#20211](#20211)) ([d8d3e16](d8d3e16)) * **bigtable:** Add Session lifecycle (Start, Close, ForceClose, readLoop, heartBeatLoop) ([#20215](#20215)) ([b9e53c6](b9e53c6)) * **bigtable:** Add Session struct + state machine ([#20117](#20117)) ([09acbb3](09acbb3)) * **bigtable:** Add session.Config.EnableDebug to gate sessionz debug state ([#20247](#20247)) ([ce74c31](ce74c31)) * **bigtable:** Add SessionClient + SessionTable + lazyPool ([#20228](#20228)) ([ab2c96c](ab2c96c)) * **bigtable:** Add SessionPoolImpl (two-tier pool + scaling + debug) ([#20225](#20225)) ([683eda8](683eda8)) * **bigtable:** Rename session pool display to <resource-id>-<PERM> ([#20248](#20248)) ([35e146e](35e146e)) * **bigtable:** Route Client.Open()-returned *Table through the Diverter ([#20273](#20273)) ([2b81c7d](2b81c7d)) * **bigtable:** State-based classification for abnormal session close ([#20243](#20243)) ([f2905b7](f2905b7)) * **bigtable:** TableShim fallback to classic on session UNIMPLEMENTED ([#20269](#20269)) ([36540af](36540af)) * **bigtable:** TTL-on-idle cache for per-resource session.TableAPI ([#20263](#20263)) ([00b2a49](00b2a49)) * **bigtable:** Wire Diverter on Client and route Open* via TableShim ([#20256](#20256)) ([b32fbd7](b32fbd7)) ### Bug Fixes * **bigtable:** AFE picker latency signal — subtract poolWait and compute TransportLatency = wire − backend at source ([#20281](#20281)) ([bb8c4d5](bb8c4d5)) * **bigtable:** Guard NewStream OnFinish against grpc-go double-fire ([#20295](#20295)) ([b51da29](b51da29)) * **bigtable:** Real per-resource pool teardown on sessionTable.Close + cache close-race gate ([#20264](#20264)) ([599aea9](599aea9)) * **bigtable:** Session.durations / session.uptime — set explicit histogram bucket boundaries ([#20276](#20276)) ([97eee22](97eee22)) * **bigtable:** SessionTableHandle self-heals across cache eviction ([#20296](#20296)) ([0dd98cd](0dd98cd)) * **bigtable:** Translate ctx errors to gRPC status on session vRPC ([#20299](#20299)) ([0f3b2a5](0f3b2a5)) * **bigtable:** Treat PingAndWarm NotFound as a successful prime ([#20219](#20219)) ([a1557ad](a1557ad)) ### Performance Improvements * **bigtable:** Delete periodic Tick loop; sizing is event-driven ([#20285](#20285)) ([2c096bd](2c096bd)) * **bigtable:** Drop pick_lost_race debug tag from CheckoutSession hot path ([#20280](#20280)) ([bd0e400](bd0e400)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com>
Summary
The OTel SDK default explicit-bucket boundaries cap at 10s
(
{0, 5, 10, 25, 50, ..., 5000, 7500, 10000}). Session lifetimes andper-session uptimes are minutes-to-hours (an idle session lives until
server-driven scale-down, typically 4+ minutes). Every sample past 10s
lands in the last bucket, so p50/p95/p99 all render as
10000msonCloud Monitoring dashboards — the metric silently pins.
Same bug applied to
session.durations,session.uptime, andsession.open_latencies— all three histograms were registeredwithout explicit bucket boundaries.
Fix
Register both hour-scale histograms with explicit boundaries:
{0}+ geometric doubling from 1ms to ~17.5min. Matches thecross-language client convention for session-scale buckets.
Introduce
SessionLifetimeBoundsnext to the existingFineGrainLatencyBoundsso future additions have a clear pick between"sub-ms latency" (
FineGrain) and "session-scale lifetime"(
SessionLifetime) sets.session.open_latenciesgetsFineGrainLatencyBoundshere too — sameclip-at-10s hazard, smaller visible impact since pre-open latency past
10s is uncommon. Included for consistency with the other three session
histograms and with
transport_latencies(which already had boundsset).
Verification
go build ./...+go vet ./internal/transport/clean.n=74 · p50=4m21s · p95=4m50s · p99=4m54s(correct, via internal HDR histogram) while the OTelsession.durationsmetric reportedp50/p95/p99 = 10000msfor the same window. After the fix, the OTel metric should display the true percentiles.Test plan
session.durationsp50/p95/p99 match sessionz "Session lifetimes" on the same sandbox runsession.uptime— same check against sessionz's uptime samplestransport_latenciesunaffected (already had explicit bounds; this change doesn't touch that call site)