feat(source/bigquery): add apiEndpoint field to override BigQuery API host - #3437
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
There was a problem hiding this comment.
Code Review
This pull request adds support for overriding the BigQuery API endpoint via a new apiEndpoint configuration option (and BIGQUERY_ENDPOINT environment variable), allowing users to route traffic through a proxy or local emulator. The changes span documentation, prebuilt configurations, and the BigQuery source implementation, including a new NormalizeEndpoint utility and corresponding tests. Feedback on the changes highlights a critical issue in NormalizeEndpoint where stripping trailing slashes from custom paths breaks relative URL resolution in the Google API client library. The reviewer provided code suggestions to preserve trailing slashes for custom paths and to update the test suite accordingly.
ca6f6b6 to
392fdbd
Compare
… host Adds an optional `apiEndpoint` config field (and `BIGQUERY_ENDPOINT` prebuilt env var) that wires `option.WithEndpoint` into both the high-level and REST BigQuery clients. Enables routing traffic through a corporate proxy or local emulator (http/https both supported). Fixes googleapis#2766 🦕
Stripping trailing slashes from all paths breaks url.URL.ResolveReference for custom path prefixes (e.g. https://proxy.com/v1/). Only strip when the path is the root "/" so relative API paths resolve correctly.
7ae65f1 to
248f9a1
Compare
|
Hi @dishaprakash - following up here since this PR has been open for a couple of weeks without a review pass yet. This closes #2766 (labeled All CI checks I have visibility into are green (lint, CLA, conventional-commits, header-check). The one exception is Could someone take a look when you have a chance, and approve the integration test run? Happy to make any changes needed. Thanks! |
shobsi
left a comment
There was a problem hiding this comment.
Hi @ayheber,
Thank you for this PR! The implementation looks great, and using option.WithEndpoint() is exactly the right approach here. I did notice that some of the automated checks are currently failing. Could you take a look at the CI logs and see what needs to be fixed? Once the checks are green, we'd be happy to get this merged.
|
👍 |
|
/gcbrun |
|
/gcbrun |
…uery API host (#3437) ## Description The BigQuery source has no way to set a custom API endpoint today. This means users who need to route BigQuery traffic through a corporate proxy or a local emulator (e.g. for governance, audit, or offline testing) have no supported mechanism — setting `HTTPS_PROXY` does not work because a forward proxy tunnels TLS directly to Google and cannot inspect the request body. This PR adds an optional `apiEndpoint` config field that passes `option.WithEndpoint(...)` to both the high-level (`cloud.google.com/go/bigquery`) and REST (`google.golang.org/api/bigquery/v2`) clients. Both `http://` and `https://` schemes are supported, enabling local emulators as well as HTTPS proxies. The prebuilt config exposes this via the `BIGQUERY_ENDPOINT` environment variable. ## PR Checklist - [x] Make sure you reviewed [CONTRIBUTING.md](https://github.com/googleapis/mcp-toolbox/blob/main/CONTRIBUTING.md) - [x] Make sure to open an issue as a [bug/issue](https://github.com/googleapis/mcp-toolbox/issues/new/choose) before writing your code! - [x] Ensure the tests and linter pass - [x] Code coverage does not decrease (if any source code was changed) - [x] Appropriate docs were updated (if necessary) - [ ] Make sure to add `!` if this involve a breaking change 🛠️ Fixes #2766 🦕 --------- Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com> Co-authored-by: Yuan Teoh <yuanteoh@google.com> 4da1600
🤖 I have created a release *beep* *boop* --- ## [1.9.0](v1.8.0...v1.9.0) (2026-08-14) ### Features * **groups:** Add ttlMs and cacheScope customization to config ([#3805](#3805)) ([a5d4947](a5d4947)) * **migrate:** Convert toolset to group kind during migration ([#3704](#3704)) ([0adeaa5](0adeaa5)) * **server/mcp:** Introduce generic client extension registry ([#3723](#3723)) ([016245c](016245c)) * **skill:** Add review-prs skill for mcp-toolbox ([#3743](#3743)) ([5b7bacc](5b7bacc)) * **source/bigquery:** Add apiEndpoint field to override BigQuery API host ([#3437](#3437)) ([4da1600](4da1600)) * **source/databaseinsights:** Add databaseinsights source ([#3461](#3461)) ([3b9615d](3b9615d)) * **sources/spanner:** Rename execute_sql_dql to execute_sql_readonly ([#3776](#3776)) ([cf5a0c8](cf5a0c8)) * **tools/bigtable:** Add admin lifecycle and listing tools ([#3596](#3596)) ([801d589](801d589)) * **tools/bigtable:** Bigtable-list-schemas MCP tool ([#3683](#3683)) ([9228c61](9228c61)) * **tools/databaseinsights:** Add Advanced Query Insights tools for AlloyDB ([#3722](#3722)) ([74d18ae](74d18ae)) * **tools/looker:** Add additional tools to allow dashboards to be modified, and their layouts altered. ([#3597](#3597)) ([b2b80fb](b2b80fb)) * **tools:** Add cloud-sql-connect-gce for pg, mysql, mssql ([#3740](#3740)) ([ca58fa4](ca58fa4)) ### Bug Fixes * **auth/mcp:** Derive PRM URL from Toolbox URL ([#3765](#3765)) ([aa30842](aa30842)) * **config:** Ignore environment variables in YAML comments ([#3807](#3807)) ([79aa732](79aa732)), refs [#3793](#3793) * **mcp:** Return Tool execution error for invalid input param ([#3799](#3799)) ([8120197](8120197)) * **prebuilt/cloud-storage:** Declare tool collections as groups ([#3764](#3764)) ([7d468be](7d468be)) * **server/mcp:** Disallow client overriding URL bound parameters ([#3798](#3798)) ([f15a9c7](f15a9c7)) * **server:** Avoid a nil-flusher panic in the SSE handler ([#3520](#3520)) ([947f42f](947f42f)) * **tools/bigquery:** Keep the provider error classification in bigquery-execute-sql ([#3738](#3738)) ([42570b8](42570b8)) * **tools/looker:** Scope the filters quoting rule to values in query description ([#3788](#3788)) ([78eb0b8](78eb0b8)) * **util:** Convert exponent-form JSON numbers in ConvertNumbers ([#3730](#3730)) ([e9713ee](e9713ee)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com>
🤖 I have created a release *beep* *boop* --- ## [1.9.0](v1.8.0...v1.9.0) (2026-08-14) ### Features * **groups:** Add ttlMs and cacheScope customization to config ([#3805](#3805)) ([a5d4947](a5d4947)) * **migrate:** Convert toolset to group kind during migration ([#3704](#3704)) ([0adeaa5](0adeaa5)) * **server/mcp:** Introduce generic client extension registry ([#3723](#3723)) ([016245c](016245c)) * **skill:** Add review-prs skill for mcp-toolbox ([#3743](#3743)) ([5b7bacc](5b7bacc)) * **source/bigquery:** Add apiEndpoint field to override BigQuery API host ([#3437](#3437)) ([4da1600](4da1600)) * **source/databaseinsights:** Add databaseinsights source ([#3461](#3461)) ([3b9615d](3b9615d)) * **sources/spanner:** Rename execute_sql_dql to execute_sql_readonly ([#3776](#3776)) ([cf5a0c8](cf5a0c8)) * **tools/bigtable:** Add admin lifecycle and listing tools ([#3596](#3596)) ([801d589](801d589)) * **tools/bigtable:** Bigtable-list-schemas MCP tool ([#3683](#3683)) ([9228c61](9228c61)) * **tools/databaseinsights:** Add Advanced Query Insights tools for AlloyDB ([#3722](#3722)) ([74d18ae](74d18ae)) * **tools/looker:** Add additional tools to allow dashboards to be modified, and their layouts altered. ([#3597](#3597)) ([b2b80fb](b2b80fb)) * **tools:** Add cloud-sql-connect-gce for pg, mysql, mssql ([#3740](#3740)) ([ca58fa4](ca58fa4)) ### Bug Fixes * **auth/mcp:** Derive PRM URL from Toolbox URL ([#3765](#3765)) ([aa30842](aa30842)) * **config:** Ignore environment variables in YAML comments ([#3807](#3807)) ([79aa732](79aa732)), refs [#3793](#3793) * **mcp:** Return Tool execution error for invalid input param ([#3799](#3799)) ([8120197](8120197)) * **prebuilt/cloud-storage:** Declare tool collections as groups ([#3764](#3764)) ([7d468be](7d468be)) * **server/mcp:** Disallow client overriding URL bound parameters ([#3798](#3798)) ([f15a9c7](f15a9c7)) * **server:** Avoid a nil-flusher panic in the SSE handler ([#3520](#3520)) ([947f42f](947f42f)) * **tools/bigquery:** Keep the provider error classification in bigquery-execute-sql ([#3738](#3738)) ([42570b8](42570b8)) * **tools/looker:** Scope the filters quoting rule to values in query description ([#3788](#3788)) ([78eb0b8](78eb0b8)) * **util:** Convert exponent-form JSON numbers in ConvertNumbers ([#3730](#3730)) ([e9713ee](e9713ee)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com> 5de8f13
🤖 I have created a release *beep* *boop* --- ## [1.9.0](googleapis/mcp-toolbox@v1.8.0...v1.9.0) (2026-08-14) ### Features * **groups:** Add ttlMs and cacheScope customization to config ([googleapis#3805](googleapis#3805)) ([a5d4947](googleapis@a5d4947)) * **migrate:** Convert toolset to group kind during migration ([googleapis#3704](googleapis#3704)) ([0adeaa5](googleapis@0adeaa5)) * **server/mcp:** Introduce generic client extension registry ([googleapis#3723](googleapis#3723)) ([016245c](googleapis@016245c)) * **skill:** Add review-prs skill for mcp-toolbox ([googleapis#3743](googleapis#3743)) ([5b7bacc](googleapis@5b7bacc)) * **source/bigquery:** Add apiEndpoint field to override BigQuery API host ([googleapis#3437](googleapis#3437)) ([4da1600](googleapis@4da1600)) * **source/databaseinsights:** Add databaseinsights source ([googleapis#3461](googleapis#3461)) ([3b9615d](googleapis@3b9615d)) * **sources/spanner:** Rename execute_sql_dql to execute_sql_readonly ([googleapis#3776](googleapis#3776)) ([cf5a0c8](googleapis@cf5a0c8)) * **tools/bigtable:** Add admin lifecycle and listing tools ([googleapis#3596](googleapis#3596)) ([801d589](googleapis@801d589)) * **tools/bigtable:** Bigtable-list-schemas MCP tool ([googleapis#3683](googleapis#3683)) ([9228c61](googleapis@9228c61)) * **tools/databaseinsights:** Add Advanced Query Insights tools for AlloyDB ([googleapis#3722](googleapis#3722)) ([74d18ae](googleapis@74d18ae)) * **tools/looker:** Add additional tools to allow dashboards to be modified, and their layouts altered. ([googleapis#3597](googleapis#3597)) ([b2b80fb](googleapis@b2b80fb)) * **tools:** Add cloud-sql-connect-gce for pg, mysql, mssql ([googleapis#3740](googleapis#3740)) ([ca58fa4](googleapis@ca58fa4)) ### Bug Fixes * **auth/mcp:** Derive PRM URL from Toolbox URL ([googleapis#3765](googleapis#3765)) ([aa30842](googleapis@aa30842)) * **config:** Ignore environment variables in YAML comments ([googleapis#3807](googleapis#3807)) ([79aa732](googleapis@79aa732)), refs [googleapis#3793](googleapis#3793) * **mcp:** Return Tool execution error for invalid input param ([googleapis#3799](googleapis#3799)) ([8120197](googleapis@8120197)) * **prebuilt/cloud-storage:** Declare tool collections as groups ([googleapis#3764](googleapis#3764)) ([7d468be](googleapis@7d468be)) * **server/mcp:** Disallow client overriding URL bound parameters ([googleapis#3798](googleapis#3798)) ([f15a9c7](googleapis@f15a9c7)) * **server:** Avoid a nil-flusher panic in the SSE handler ([googleapis#3520](googleapis#3520)) ([947f42f](googleapis@947f42f)) * **tools/bigquery:** Keep the provider error classification in bigquery-execute-sql ([googleapis#3738](googleapis#3738)) ([42570b8](googleapis@42570b8)) * **tools/looker:** Scope the filters quoting rule to values in query description ([googleapis#3788](googleapis#3788)) ([78eb0b8](googleapis@78eb0b8)) * **util:** Convert exponent-form JSON numbers in ConvertNumbers ([googleapis#3730](googleapis#3730)) ([e9713ee](googleapis@e9713ee)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com> 5de8f13
🤖 I have created a release *beep* *boop* --- ## [1.9.0](googleapis/mcp-toolbox@v1.8.0...v1.9.0) (2026-08-14) ### Features * **groups:** Add ttlMs and cacheScope customization to config ([googleapis#3805](googleapis#3805)) ([a5d4947](googleapis@a5d4947)) * **migrate:** Convert toolset to group kind during migration ([googleapis#3704](googleapis#3704)) ([0adeaa5](googleapis@0adeaa5)) * **server/mcp:** Introduce generic client extension registry ([googleapis#3723](googleapis#3723)) ([016245c](googleapis@016245c)) * **skill:** Add review-prs skill for mcp-toolbox ([googleapis#3743](googleapis#3743)) ([5b7bacc](googleapis@5b7bacc)) * **source/bigquery:** Add apiEndpoint field to override BigQuery API host ([googleapis#3437](googleapis#3437)) ([4da1600](googleapis@4da1600)) * **source/databaseinsights:** Add databaseinsights source ([googleapis#3461](googleapis#3461)) ([3b9615d](googleapis@3b9615d)) * **sources/spanner:** Rename execute_sql_dql to execute_sql_readonly ([googleapis#3776](googleapis#3776)) ([cf5a0c8](googleapis@cf5a0c8)) * **tools/bigtable:** Add admin lifecycle and listing tools ([googleapis#3596](googleapis#3596)) ([801d589](googleapis@801d589)) * **tools/bigtable:** Bigtable-list-schemas MCP tool ([googleapis#3683](googleapis#3683)) ([9228c61](googleapis@9228c61)) * **tools/databaseinsights:** Add Advanced Query Insights tools for AlloyDB ([googleapis#3722](googleapis#3722)) ([74d18ae](googleapis@74d18ae)) * **tools/looker:** Add additional tools to allow dashboards to be modified, and their layouts altered. ([googleapis#3597](googleapis#3597)) ([b2b80fb](googleapis@b2b80fb)) * **tools:** Add cloud-sql-connect-gce for pg, mysql, mssql ([googleapis#3740](googleapis#3740)) ([ca58fa4](googleapis@ca58fa4)) ### Bug Fixes * **auth/mcp:** Derive PRM URL from Toolbox URL ([googleapis#3765](googleapis#3765)) ([aa30842](googleapis@aa30842)) * **config:** Ignore environment variables in YAML comments ([googleapis#3807](googleapis#3807)) ([79aa732](googleapis@79aa732)), refs [googleapis#3793](googleapis#3793) * **mcp:** Return Tool execution error for invalid input param ([googleapis#3799](googleapis#3799)) ([8120197](googleapis@8120197)) * **prebuilt/cloud-storage:** Declare tool collections as groups ([googleapis#3764](googleapis#3764)) ([7d468be](googleapis@7d468be)) * **server/mcp:** Disallow client overriding URL bound parameters ([googleapis#3798](googleapis#3798)) ([f15a9c7](googleapis@f15a9c7)) * **server:** Avoid a nil-flusher panic in the SSE handler ([googleapis#3520](googleapis#3520)) ([947f42f](googleapis@947f42f)) * **tools/bigquery:** Keep the provider error classification in bigquery-execute-sql ([googleapis#3738](googleapis#3738)) ([42570b8](googleapis@42570b8)) * **tools/looker:** Scope the filters quoting rule to values in query description ([googleapis#3788](googleapis#3788)) ([78eb0b8](googleapis@78eb0b8)) * **util:** Convert exponent-form JSON numbers in ConvertNumbers ([googleapis#3730](googleapis#3730)) ([e9713ee](googleapis@e9713ee)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com> 5de8f13
🤖 I have created a release *beep* *boop* --- ## [1.9.0](googleapis/mcp-toolbox@v1.8.0...v1.9.0) (2026-08-14) ### Features * **groups:** Add ttlMs and cacheScope customization to config ([googleapis#3805](googleapis#3805)) ([a5d4947](googleapis@a5d4947)) * **migrate:** Convert toolset to group kind during migration ([googleapis#3704](googleapis#3704)) ([0adeaa5](googleapis@0adeaa5)) * **server/mcp:** Introduce generic client extension registry ([googleapis#3723](googleapis#3723)) ([016245c](googleapis@016245c)) * **skill:** Add review-prs skill for mcp-toolbox ([googleapis#3743](googleapis#3743)) ([5b7bacc](googleapis@5b7bacc)) * **source/bigquery:** Add apiEndpoint field to override BigQuery API host ([googleapis#3437](googleapis#3437)) ([4da1600](googleapis@4da1600)) * **source/databaseinsights:** Add databaseinsights source ([googleapis#3461](googleapis#3461)) ([3b9615d](googleapis@3b9615d)) * **sources/spanner:** Rename execute_sql_dql to execute_sql_readonly ([googleapis#3776](googleapis#3776)) ([cf5a0c8](googleapis@cf5a0c8)) * **tools/bigtable:** Add admin lifecycle and listing tools ([googleapis#3596](googleapis#3596)) ([801d589](googleapis@801d589)) * **tools/bigtable:** Bigtable-list-schemas MCP tool ([googleapis#3683](googleapis#3683)) ([9228c61](googleapis@9228c61)) * **tools/databaseinsights:** Add Advanced Query Insights tools for AlloyDB ([googleapis#3722](googleapis#3722)) ([74d18ae](googleapis@74d18ae)) * **tools/looker:** Add additional tools to allow dashboards to be modified, and their layouts altered. ([googleapis#3597](googleapis#3597)) ([b2b80fb](googleapis@b2b80fb)) * **tools:** Add cloud-sql-connect-gce for pg, mysql, mssql ([googleapis#3740](googleapis#3740)) ([ca58fa4](googleapis@ca58fa4)) ### Bug Fixes * **auth/mcp:** Derive PRM URL from Toolbox URL ([googleapis#3765](googleapis#3765)) ([aa30842](googleapis@aa30842)) * **config:** Ignore environment variables in YAML comments ([googleapis#3807](googleapis#3807)) ([79aa732](googleapis@79aa732)), refs [googleapis#3793](googleapis#3793) * **mcp:** Return Tool execution error for invalid input param ([googleapis#3799](googleapis#3799)) ([8120197](googleapis@8120197)) * **prebuilt/cloud-storage:** Declare tool collections as groups ([googleapis#3764](googleapis#3764)) ([7d468be](googleapis@7d468be)) * **server/mcp:** Disallow client overriding URL bound parameters ([googleapis#3798](googleapis#3798)) ([f15a9c7](googleapis@f15a9c7)) * **server:** Avoid a nil-flusher panic in the SSE handler ([googleapis#3520](googleapis#3520)) ([947f42f](googleapis@947f42f)) * **tools/bigquery:** Keep the provider error classification in bigquery-execute-sql ([googleapis#3738](googleapis#3738)) ([42570b8](googleapis@42570b8)) * **tools/looker:** Scope the filters quoting rule to values in query description ([googleapis#3788](googleapis#3788)) ([78eb0b8](googleapis@78eb0b8)) * **util:** Convert exponent-form JSON numbers in ConvertNumbers ([googleapis#3730](googleapis#3730)) ([e9713ee](googleapis@e9713ee)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). --------- Co-authored-by: release-please[bot] <55107282+release-please[bot]@users.noreply.github.com> Co-authored-by: Yuan Teoh <45984206+Yuan325@users.noreply.github.com> 5de8f13
Description
The BigQuery source has no way to set a custom API endpoint today. This
means users who need to route BigQuery traffic through a corporate proxy
or a local emulator (e.g. for governance, audit, or offline testing) have
no supported mechanism — setting
HTTPS_PROXYdoes not work because aforward proxy tunnels TLS directly to Google and cannot inspect the
request body.
This PR adds an optional
apiEndpointconfig field that passesoption.WithEndpoint(...)to both the high-level (cloud.google.com/go/bigquery)and REST (
google.golang.org/api/bigquery/v2) clients. Bothhttp://andhttps://schemes are supported, enabling local emulators as well asHTTPS proxies. The prebuilt config exposes this via the
BIGQUERY_ENDPOINTenvironment variable.
PR Checklist
!if this involve a breaking change🛠️ Fixes #2766 🦕