Harden permissions for persisted secret files - #23465
Conversation
…s and FIFOs fopen_secret_write() hard-failed when fchmod() failed, but fchmod() is gated on ownership, not on write access. A secret left behind by an agent that ran as root stays writable through the netdata group while being un-chmod-able by the netdata user, and some filesystems refuse chmod outright. Claiming, claimed_id persistence and bearer token persistence would then fail permanently where they used to work. Tolerate the failure when the file is already inaccessible to group and other, and log a warning; keep failing when it is not. The secret directories are group-writable (cloud.d is 0770), so the final path component is attacker-controllable. Add O_NOFOLLOW so a planted symlink is not chmod-ed and truncated, and fstat()/S_ISREG() plus O_NONBLOCK so a planted FIFO neither blocks the open nor receives the secret. Also convert dyncfg_file_save(): dyncfg payloads are user-submitted job configurations that routinely carry database passwords, SNMP communities and API tokens, and were landing group-readable under the daemon's umask. Tests cover the symlink and FIFO refusals, including that the symlink target's mode and contents are left untouched.
…e write Refusing to write when fchmod() fails left claiming permanently broken wherever a secret was left behind by an agent that ran as root: such a file is writable through the netdata group but cannot be tightened, because fchmod() is gated on ownership. Verified in docker with a non-root daemon and a pre-existing 0660 root:netdata cloud.conf/claimed_id/private.pem - claiming failed with "Failed to save claiming info to disk" and "Cannot write private key file". The secret directories are group-writable (cloud.d is 0770), so when the file is accessible to group or other and cannot be chmod-ed, unlink it and recreate it with O_CREAT|O_EXCL at 0600. This is a truncating open, so the old content was being discarded anyway, and O_EXCL makes a path recreated in between an error rather than a place to write a secret into. When the unlink is refused the write still fails, so a secret is never written to a file that could not be made private. The same scenarios now claim successfully and leave every secret at 0600, owned by the daemon user. Also drop two hand-rolled copies of this logic now that the helper covers them: the MCP dev-preview key writer (which deleted the key when fchmod() failed) and the management API key writer. The latter keeps its mkstemp()+rename() path on platforms without O_NOFOLLOW, since that is where it gets its symlink safety.
The symlink refusal was guarded by #ifdef O_NOFOLLOW while the helper's contract promised it unconditionally, so on the Windows build (mingw/UCRT does not define the flag) open() followed a planted symlink and the fstat() only confirmed that the symlink's target was a regular file. Every caller was affected. Check the path explicitly where the flag is missing, the way status-file-io.c already does: lstat() first and refuse a symlink with ELOOP (the errno O_NOFOLLOW would have produced) or any other non-regular file with EINVAL; add O_EXCL when nothing is there, so a path planted between the lstat() and the open() fails with EEXIST instead of being followed; and compare the opened inode against what was inspected to catch a swap. The remaining gap - a hardlink planted before the lstat(), normally blocked by fs.protected_hardlinks - is now stated in the contract instead of being implied away. All of it sits in #else / #ifndef blocks, so platforms with O_NOFOLLOW are unaffected. The management API key writer keeps its own mkstemp()+rename() path on those platforms, since the atomic rename also closes the hardlink window; the reason is now recorded there.
…e is refused Review follow-ups on the secret-write helper: - When a file is exposed and un-chmod-able and the replacement unlink() also fails, report unlink()'s errno instead of the earlier fchmod() one, and log both. The unlink error is the actionable one: a secret bind-mounted as a single file gives EBUSY (verified), a read-only directory gives EROFS. - fdopen() before ftruncate(), so a failure there leaves the previous content in place like every other refusal in this helper does, instead of leaving an empty file behind while reporting failure. - Honour a "b" in the caller's mode when the platform has O_BINARY: the descriptor decides newline translation there, and the PEM writers pass "wb". - Guard S_ISLNK, which had not been used before in code built for Windows. A symlink fails the following S_ISREG() check anyway, so the refusal holds either way and only the reported errno differs. - Correct the documented contract: an already-owner-only file on a chmod-refusing filesystem may keep a mode tighter than 0600, and the lstat()/O_EXCL/inode fallback closes the symlink redirect but not a hardlink planted before it - which is why the management API key writer keeps its own rename() path.
…them The checks lived only in a standalone EXCLUDE_FROM_ALL target that no workflow builds, so nothing verified fopen_secret_write() automatically. Move them into fopen_secret_write_unittest() next to the implementation, where paths_unittest() runs them - and paths_unittest() is already executed in CI by `netdata -W unittest` through tests/run-unit-tests.sh. The standalone target stays as a thin runner for the same function, so the checks exist in one place instead of being duplicated. Skipped on Windows, where POSIX mode bits, symlinks and FIFOs do not carry their intended meaning, and the process umask is saved and restored so the surrounding unittests are unaffected.
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
3 issues found across 12 files
Confidence score: 2/5
- In
src/libnetdata/paths/paths.h, the current secret-file API can still expose secrets via pre-existing hardlinks in group-writable directories, which is a direct confidentiality risk even if callers try to set stricter modes later — switch to creating a0600temp file and atomicallyrename()it into place (or enforce that pattern for all callers). - In
src/web/api/v1/api_v1_manage.c, the successful existing-key read path skips the hardening path, so legacy0660keys remain group-readable and continue leaking sensitive key material — enforce0600on the success path before returning the key. - In
src/claim/claim-with-api.c, upgrade flows that find an existingpublic.pemskip the write path, leaving olderprivate.pempermissions too broad and creating persistent exposure after migration — harden both existing-key paths before the completion-marker return using the non-truncating secure write/replace pattern.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/web/api/v1/api_v1_manage.c">
<violation number="1" location="src/web/api/v1/api_v1_manage.c:57">
P1: When an existing key is valid, this call is never reached, so legacy `0660` keys remain group-readable. Enforce `0600` on the successful read path before returning the key.</violation>
</file>
<file name="src/libnetdata/paths/paths.h">
<violation number="1" location="src/libnetdata/paths/paths.h:39">
P0: This API still leaks secrets through pre-existing hardlinks in group-writable secret directories. Create a 0600 temporary file and atomically `rename()` it into place, or require every caller to use that pattern instead of this helper.
(Based on your team's feedback about atomic private file writes.)</violation>
</file>
<file name="src/claim/claim-with-api.c">
<violation number="1" location="src/claim/claim-with-api.c:62">
P1: When an existing `public.pem` is present after an upgrade, this write is skipped, so an older `private.pem` remains group-readable. Harden both existing key paths before the completion-marker return, using a non-truncating regular-file-safe operation.</violation>
</file>
Architecture diagram
sequenceDiagram
participant Caller as Secret File Callers
participant Helper as fopen_secret_write()
participant OpenFD as fopen_secret_open_fd()
participant FS as Filesystem
participant Test as Unittest Suite
Note over Caller,FS: Secret File Write Flow (all callers now route through helper)
Caller->>Helper: fopen_secret_write(path, "w")
Helper->>OpenFD: fopen_secret_open_fd(path, mode, 0, &replace_it)
OpenFD->>OpenFD: Build flags (O_WRONLY|O_CREAT|O_CLOEXEC)
alt Platform has O_NOFOLLOW
OpenFD->>OpenFD: Add O_NOFOLLOW flag
else No O_NOFOLLOW (Windows)
OpenFD->>FS: lstat(path)
alt Path exists
alt Is symlink
OpenFD-->>Helper: ELOOP error
Helper-->>Caller: NULL
else Not regular file
OpenFD-->>Helper: EINVAL error
Helper-->>Caller: NULL
else Regular file
OpenFD->>OpenFD: Record inode identity
end
else Path missing (ENOENT)
OpenFD->>OpenFD: Add O_EXCL flag
end
end
OpenFD->>OpenFD: Add O_NONBLOCK flag
opt Mode contains "b"
OpenFD->>OpenFD: Add O_BINARY flag
end
OpenFD->>FS: open(path, flags, 0600)
alt Open fails
FS-->>OpenFD: -1 with errno
OpenFD-->>Helper: -1
Helper-->>Caller: NULL
else Open succeeds
FS-->>OpenFD: fd
OpenFD->>FS: fstat(fd)
alt Not regular file (FIFO/device)
OpenFD-->>Helper: EINVAL error
Helper-->>Caller: NULL
else No O_NOFOLLOW and inode changed
OpenFD-->>Helper: ELOOP error
Helper-->>Caller: NULL
else Regular file confirmed
OpenFD->>FS: fchmod(fd, 0600)
alt fchmod succeeds
OpenFD-->>Helper: fd with 0600 mode
else fchmod fails
alt File group/other accessible
OpenFD-->>Helper: -1, set replace_it=true
Helper->>FS: unlink(path)
alt Unlink succeeds
Helper->>OpenFD: Retry with O_EXCL flag
OpenFD->>FS: open(path, flags|O_EXCL, 0600)
FS-->>OpenFD: new fd
OpenFD-->>Helper: fd
else Unlink fails
FS-->>Helper: errno (EBUSY/EROFS)
Helper-->>Caller: NULL (refuse to write)
end
else Already owner-only
OpenFD-->>Helper: fd (continue with warning)
end
end
end
end
Helper->>FS: fdopen(fd, mode)
alt fdopen fails
Helper-->>Caller: NULL (previous content preserved)
else fdopen succeeds
Helper->>FS: ftruncate(fd, 0)
alt Truncate fails
Helper-->>Caller: NULL (previous content preserved)
else Truncate succeeds
Helper-->>Caller: FILE* (truncated, mode 0600, safe)
end
end
Note over Test: Unit Test Verification (runs under umask 0022)
Test->>Helper: fopen_secret_write_unittest()
Test->>Test: Set umask(0022)
Test->>Helper: fopen_secret_write(created, "w")
Helper->>Test: FILE*
Test->>Test: Write and close
Test->>FS: stat(created)
alt Mode is 0600
FS-->>Test: PASS
else Mode differs
FS-->>Test: FAIL (proves fchmod worked)
end
Test->>FS: Control: plain fopen() produces 0644
Test->>FS: Symlink victim check
Test->>FS: FIFO victim check
Test->>Test: Restore umask
Test-->>Test: Return error count
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| // to a file this call could not make private. | ||
| // | ||
| // Returns NULL and leaves errno set on failure. | ||
| FILE *fopen_secret_write(const char *filename, const char *mode); |
There was a problem hiding this comment.
P0: This API still leaks secrets through pre-existing hardlinks in group-writable secret directories. Create a 0600 temporary file and atomically rename() it into place, or require every caller to use that pattern instead of this helper.
(Based on your team's feedback about atomic private file writes.)
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/libnetdata/paths/paths.h, line 39:
<comment>This API still leaks secrets through pre-existing hardlinks in group-writable secret directories. Create a 0600 temporary file and atomically `rename()` it into place, or require every caller to use that pattern instead of this helper.
(Based on your team's feedback about atomic private file writes.) </comment>
<file context>
@@ -11,6 +11,33 @@ char *filename_from_path_entry_strdupz(const char *path, const char *entry);
+// to a file this call could not make private.
+//
+// Returns NULL and leaves errno set on failure.
+FILE *fopen_secret_write(const char *filename, const char *mode);
+
bool path_entry_is_file(const char *path, const char *entry);
</file context>
| fd = open(api_key_filename, O_RDWR|O_CREAT|O_CLOEXEC|O_NONBLOCK|O_NOFOLLOW, 0600); | ||
| if(fd == -1) { | ||
| // fopen_secret_write() owns the symlink, regular-file and 0600 guarantees | ||
| FILE *fp = fopen_secret_write(api_key_filename, "w"); |
There was a problem hiding this comment.
P1: When an existing key is valid, this call is never reached, so legacy 0660 keys remain group-readable. Enforce 0600 on the successful read path before returning the key.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/web/api/v1/api_v1_manage.c, line 57:
<comment>When an existing key is valid, this call is never reached, so legacy `0660` keys remain group-readable. Enforce `0600` on the successful read path before returning the key.</comment>
<file context>
@@ -53,43 +53,28 @@ static char *get_mgmt_api_key(void) {
- fd = open(api_key_filename, O_RDWR|O_CREAT|O_CLOEXEC|O_NONBLOCK|O_NOFOLLOW, 0600);
- if(fd == -1) {
+ // fopen_secret_write() owns the symlink, regular-file and 0600 guarantees
+ FILE *fp = fopen_secret_write(api_key_filename, "w");
+ if(!fp) {
netdata_log_error("Cannot create unique management API key file '%s'. Please adjust config parameter 'netdata management api key file' to a proper path and file.", api_key_filename);
</file context>
|
|
||
| // Save private key | ||
| fp = fopen(private_key_file, "wb"); | ||
| fp = fopen_secret_write(private_key_file, "wb"); |
There was a problem hiding this comment.
P1: When an existing public.pem is present after an upgrade, this write is skipped, so an older private.pem remains group-readable. Harden both existing key paths before the completion-marker return, using a non-truncating regular-file-safe operation.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/claim/claim-with-api.c, line 62:
<comment>When an existing `public.pem` is present after an upgrade, this write is skipped, so an older `private.pem` remains group-readable. Harden both existing key paths before the completion-marker return, using a non-truncating regular-file-safe operation.</comment>
<file context>
@@ -59,7 +59,7 @@ static bool check_and_generate_certificates() {
// Save private key
- fp = fopen(private_key_file, "wb");
+ fp = fopen_secret_write(private_key_file, "wb");
if (!fp || !PEM_write_PrivateKey(fp, pkey, NULL, NULL, 0, NULL, NULL)) {
claim_agent_failure_reason_set("Cannot write private key file: %s", private_key_file);
</file context>
…gacy files Two reachable holes, both from trusting a file's mode alone: The helper accepted any pre-existing file whose mode was owner-only, without checking who owns it. A daemon running as root - the default in many containers - can write to a file owned by anyone, so a local user able to create files in the group-writable cloud.d could pre-create the path at 0600, have the secret written into an inode they own, then chmod it back and read it. Verified in a container: the secret was recovered by its owner afterwards. The same blind spot covered a pre-existing hardlink, which exposes the bytes under a name we do not control. A pre-existing file is now only written to when it is ours, has a single link, and is already inaccessible to group and other; anything else is replaced by a fresh file. Replacement is also preferred over tightening a wide file in place, since a reader may already hold the old inode open - but when the file cannot be replaced (EBUSY for a single-file bind mount, EROFS for a read-only directory) and it is provably ours with one link, it is still tightened in place rather than failing a write that used to work. The second hole is that all of this only applies when a file is written, and the files in cloud.d are written once and then only read: an agent claimed by an earlier release keeps them at 0660 forever, because every claim entry point is gated behind claimed_id being empty and never runs again. secret_file_harden() tightens them in place at every startup instead. It is deliberately chmod-only - Netdata Cloud holds the matching public key, so replacing the keypair would break this node's identity with no way back short of re-claiming. Verified end-to-end in containers: an already-claimed agent whose cloud.d files are 0660 now reaches 0600 with content and identity untouched and without re-claiming, while a file owned by another uid is replaced instead of written into.
| // a file an older agent left at 0660 must be tightened, and a shorter secret | ||
| // must not leave a tail of the longer one behind | ||
| fp = fopen(preexisting, "w"); | ||
| if(!fp || fprintf(fp, "a-long-previous-secret") < 0 || fclose(fp) != 0 || chmod(preexisting, 0660) != 0) { |
…very boot systemd-tmpfiles-setup.service runs `systemd-tmpfiles --create --remove --boot` at every boot, and a `z` line applies its mode to files that already exist. The cloud.d rule set 0660, so the claim private key, the claim token in cloud.conf and claimed_id were handed back to the netdata group on every reboot no matter what the agent wrote them as - and the installer re-applies the same rules. Set it to 0600 to match what the agent itself writes, and note the coupling in a comment so the two do not drift apart again. netdata.api.key deliberately stays 0660: operators are told to read that token directly to call the health management API, so tightening it would break a documented workflow. That is now recorded next to the rule. Verified with systemd-tmpfiles --root against a temp tree: cloud.d files go 0660 -> 0600, netdata.api.key stays 0660, and the remaining rules are unchanged.
There was a problem hiding this comment.
3 issues found across 6 files (changes from recent commits).
Confidence score: 2/5
- In
src/libnetdata/paths/paths.c,secret_file_harden()can follow symlinks on systems withoutO_NOFOLLOW, sofchmod()may affect an unintended target and weaken secret-file safety boundaries; add the same pre/post-open identity checks used byfopen_secret_open_fd()(or fail hard on mismatch). - In
src/web/api/mcp_auth.c, ignoring afalsereturn fromsecret_file_harden()means MCP auth can keep running with an API key file that remains too broadly readable, leaving credentials exposed despite startup hardening; treat hardening failure as fatal for loading/using that secret. - In
src/claim/claim.c,cloud_secrets_harden()skipscloud.d/tokenandcloud.d/roomsunder split-file auto-claiming, so legacy group-readable permissions can persist and leak claim credentials; include both persisted names in the hardening pass to close the gap.
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="src/claim/claim.c">
<violation number="1" location="src/claim/claim.c:165">
P1: When split-file auto-claiming is configured, `cloud.d/token` and `cloud.d/rooms` can remain legacy group-readable because `cloud_secrets_harden()` never visits them. Add both persisted credential names so startup hardens the complete cloud.d credential set.</violation>
</file>
<file name="src/web/api/mcp_auth.c">
<violation number="1" location="src/web/api/mcp_auth.c:110">
P1: When `secret_file_harden()` cannot tighten an existing key, this call ignores its `false` result and continues with the loaded secret. MCP authentication remains enabled while the persisted API key can stay group- or other-readable; fail closed after hardening fails.</violation>
</file>
<file name="src/libnetdata/paths/paths.c">
<violation number="1" location="src/libnetdata/paths/paths.c:556">
P1: On platforms without `O_NOFOLLOW`, `secret_file_harden()` follows a symlink and applies `fchmod()` to its target. Reject symlinks with the same pre-open and post-open identity checks as `fopen_secret_open_fd()`, or fail closed when those checks are unavailable.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Re-trigger cubic
| // holds the matching public key and regenerating it would break this node's | ||
| // identity with no way back short of re-claiming. | ||
| static void cloud_secrets_harden(void) { | ||
| static const char *secrets[] = { "claimed_id", "cloud.conf", "private.pem", "public.pem" }; |
There was a problem hiding this comment.
P1: When split-file auto-claiming is configured, cloud.d/token and cloud.d/rooms can remain legacy group-readable because cloud_secrets_harden() never visits them. Add both persisted credential names so startup hardens the complete cloud.d credential set.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/claim/claim.c, line 165:
<comment>When split-file auto-claiming is configured, `cloud.d/token` and `cloud.d/rooms` can remain legacy group-readable because `cloud_secrets_harden()` never visits them. Add both persisted credential names so startup hardens the complete cloud.d credential set.</comment>
<file context>
@@ -154,7 +154,25 @@ bool claim_id_matches_any(const char *claim_id) {
+// holds the matching public key and regenerating it would break this node's
+// identity with no way back short of re-claiming.
+static void cloud_secrets_harden(void) {
+ static const char *secrets[] = { "claimed_id", "cloud.conf", "private.pem", "public.pem" };
+
+ for(size_t i = 0; i < sizeof(secrets) / sizeof(secrets[0]); i++) {
</file context>
| static const char *secrets[] = { "claimed_id", "cloud.conf", "private.pem", "public.pem" }; | |
| static const char *secrets[] = { "claimed_id", "cloud.conf", "private.pem", "public.pem", "token", "rooms" }; |
| char loaded_path[PATH_MAX]; | ||
| snprintf(loaded_path, sizeof(loaded_path), "%s/%s", | ||
| netdata_configured_varlib_dir, MCP_DEV_PREVIEW_API_KEY_FILENAME); | ||
| secret_file_harden(loaded_path); |
There was a problem hiding this comment.
P1: When secret_file_harden() cannot tighten an existing key, this call ignores its false result and continues with the loaded secret. MCP authentication remains enabled while the persisted API key can stay group- or other-readable; fail closed after hardening fails.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/web/api/mcp_auth.c, line 110:
<comment>When `secret_file_harden()` cannot tighten an existing key, this call ignores its `false` result and continues with the loaded secret. MCP authentication remains enabled while the persisted API key can stay group- or other-readable; fail closed after hardening fails.</comment>
<file context>
@@ -101,7 +101,15 @@ static bool mcp_api_key_load(void) {
+ char loaded_path[PATH_MAX];
+ snprintf(loaded_path, sizeof(loaded_path), "%s/%s",
+ netdata_configured_varlib_dir, MCP_DEV_PREVIEW_API_KEY_FILENAME);
+ secret_file_harden(loaded_path);
+ }
+ else {
</file context>
| secret_file_harden(loaded_path); | |
| if (!secret_file_harden(loaded_path)) { | |
| mcp_dev_preview_api_key[0] = '\0'; | |
| netdata_log_error("MCP: Refusing to use API key because its file permissions could not be tightened"); | |
| return; | |
| } |
| flags |= O_NONBLOCK; | ||
| #endif | ||
|
|
||
| int fd = open(filename, flags); |
There was a problem hiding this comment.
P1: On platforms without O_NOFOLLOW, secret_file_harden() follows a symlink and applies fchmod() to its target. Reject symlinks with the same pre-open and post-open identity checks as fopen_secret_open_fd(), or fail closed when those checks are unavailable.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At src/libnetdata/paths/paths.c, line 556:
<comment>On platforms without `O_NOFOLLOW`, `secret_file_harden()` follows a symlink and applies `fchmod()` to its target. Reject symlinks with the same pre-open and post-open identity checks as `fopen_secret_open_fd()`, or fail closed when those checks are unavailable.</comment>
<file context>
@@ -498,6 +538,53 @@ void recursive_config_double_dir_load(const char *user_path, const char *stock_p
+ flags |= O_NONBLOCK;
+#endif
+
+ int fd = open(filename, flags);
+ if(fd == -1)
+ return errno == ENOENT; // nothing to harden is not a failure
</file context>
|
| snprintfz(linked, sizeof(linked), "%s/linked", dir); | ||
| snprintfz(othername, sizeof(othername), "%s/othername", dir); | ||
| fp = fopen(linked, "w"); | ||
| if(!fp || fprintf(fp, "before") < 0 || fclose(fp) != 0 || chmod(linked, 0600) != 0 || link(linked, othername) != 0) { |
|
|
||
| fp = fopen(foreign, "w"); | ||
| if(!fp || fprintf(fp, "attacker") < 0 || fclose(fp) != 0 || | ||
| chown(foreign, 12345, 12345) != 0 || chmod(foreign, 0600) != 0 || stat(foreign, &st) != 0) { |
|
|
||
| fp = fopen(foreign, "w"); | ||
| if(!fp || fprintf(fp, "attacker") < 0 || fclose(fp) != 0 || | ||
| chown(foreign, 12345, 12345) != 0 || chmod(foreign, 0600) != 0 || stat(foreign, &st) != 0) { |
| char legacy[FILENAME_MAX + 1]; | ||
| snprintfz(legacy, sizeof(legacy), "%s/legacy", dir); | ||
| fp = fopen(legacy, "w"); | ||
| if(!fp || fprintf(fp, "old-secret") < 0 || fclose(fp) != 0 || chmod(legacy, 0660) != 0 || stat(legacy, &st) != 0) { |



Summary