When EnablePersistentBuffer=1 is enabled, Agent 2 can terminate the whole service after a single local SQLite write failure in the persistent buffer path.

In the investigated incident, the environmental trigger was a snapshot/VSS storage freeze or disturbance. Windows logged VSS/VMware snapshot activity, NTFS delayed-write failures, failed transaction-log flushes, and disk surprise-removal events shortly before Agent 2 failed.

That explains why SQLite returned:

disk I/O error: Access is denied.

The concern is that a transient persistent-buffer write failure is treated as a process-fatal condition.

Observed behavior:

Agent 2 log:

cannot execute SQL statement : disk I/O error: Access is denied.
Critical failure: disk I/O error: Access is denied.

Stack trace points to:

golang.zabbix.com/agent2/internal/agent/resultcache.(*DiskCache).write
golang.zabbix.com/agent2/internal/agent/resultcache.(*DiskCache).run

Windows Service Control Manager then reports that Zabbix Agent 2 terminated unexpectedly.

Source locations:

Checked against 7.0.27:

src/go/internal/agent/resultcache/diskcache.go:371 - func (c *DiskCache) write(r *plugin.Result) src/go/internal/agent/resultcache/diskcache.go:469 - logs cannot execute SQL statement : %s src/go/internal/agent/resultcache/diskcache.go:473 - calls panic(err) 
src/go/internal/agent/resultcache/diskcache.go:477 - func (c *DiskCache) writeCommand(cr *CommandResult) src/go/internal/agent/resultcache/diskcache.go:523 - logs cannot execute SQL statement : %s src/go/internal/agent/resultcache/diskcache.go:527 - calls panic(err)
src/go/internal/agent/resultcache/diskcache.go:532 - DiskCache.run() defers log.PanicHook(), which logs the critical failure and re-panics.

Current behavior in both write() and writeCommand() is therefore:

1. execute SQLite insert,
2. log SQL execution error if stmt.Exec(...) fails,
3. panic on the remaining error,
4. PanicHook() logs a critical failure and the Agent 2 process terminates.

Expected behavior

Agent 2 should handle transient persistent-buffer SQLite I/O failures more gracefully than terminating the whole service.

Possible acceptable behavior could include one or more of:

• log the persistent-buffer I/O failure as an error,
• retry local SQLite writes with backoff,
• reopen/reinitialize the SQLite handle after transient storage errors,
• temporarily mark the persistent buffer as degraded,
• continue running checks where possible,
• explicitly report that buffered data may be lost while local storage is unavailable.

Exact implementation is for You to decide. The important expectation is that one transient local persistent-buffer write error should not necessarily kill the whole Agent 2 process.

Suggested reproduction

1. Install Zabbix Agent 2.

2. Enable active checks and persistent buffer:

EnablePersistentBuffer=1
PersistentBufferFile=<local writable path>\zabbix_agent2_buffer.db

3. Configure frequent active items so Agent 2 regularly writes to the persistent buffer.

4. While Agent 2 is running, temporarily make the persistent buffer path or file unavailable/unwritable, or simulate a storage freeze that makes SQLite writes fail.

5. Wait for Agent 2 to write an active check result.

Expected result for the bug:

cannot execute SQL statement : disk I/O error: Access is denied.
Critical failure: disk I/O error: Access is denied. 

Agent 2 process terminates.

Also a note:

The issue can be reproduced at customer's environment, yet since we don't have same infrastructure, we cannot reproduce it on our premises.