[#ZBX-6036] Zabbix proxy can start if DB is down or corrupted

Comment by Oleksii Zagorskyi [ 2012 Dec 28 ]

We have similar issue for zabbix server - ~~ZBX-4611~~

Comment by Alexey Pustovalov [ 2015 Mar 25 ]

Zabbix can check sqlite3 database integrity using:

pragma integrity_check;

http://www.sqlite.org/pragma.html#pragma_integrity_check

This pragma does an integrity check of the entire database. The integrity_check pragma looks for out-of-order records, missing pages, malformed records, missing index entries, and UNIQUE and NOT NULL constraint errors. If the integrity_check pragma finds problems, strings are returned (as multiple rows with a single column per row) which describe the problems. Pragma integrity_check will return at most N errors before the analysis quits, with N defaulting to 100. If pragma integrity_check finds no errors, a single row with the value 'ok' is returned.

For example:

echo "asdasdasdasdad" > /tmp/test.db
root@ubuntu:/usr/share/zabbix# sqlite3 /tmp/test.db 
SQLite version 3.8.2 2013-12-06 14:53:30
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> pragma integrity_check;
Error: file is encrypted or is not a database
sqlite>

Comment by Alexander Vladishev [ 2015 Mar 26 ]

I close the issue as "Won't fix".

The database integrity should be checked outside Zabbix because it can be very slow operation. This can be checked by native database tools.

Comment by Chris Christensen [ 2015 Mar 26 ]

@sasha we had an issue specifically with corrupt proxy DBs (sqlite) yet all proxy checks (check-in time, running processes) were valid. One addition we added was:

{zabbix-proxy1.example.com:vfs.file.time[/path/to/zabbix_proxy.db].fuzzytime(7200)}=0

Given that the proxies work well with SQLite DB's (store-and-forward) it seems prudent to at least document how to perform this check out-of-band, as this is a bad failure case.

What's the recommended takeaway? An init.d script, additional custom userparam, ...? A built-in integrity check still seems the most optimal.

Comment by Alexander Vladishev [ 2015 Apr 09 ]

I think the best solution is a custom script because built-in check:

will work only with SQLite3 database (other databases should be checked by other way)

it can significantly increase Zabbix proxy startup time. For example on Raspberry PI:

$ ls -lh sqlite3.db 
-rw-r--r-- 1 pi pi 242M Apr  9 07:53 sqlite3.db

$ time echo 'pragma integrity_check;' | sqlite3 sqlite3.db 
ok

real	4m55.227s
user	4m44.690s
sys	0m9.750s

$ cat /etc/issue
Raspbian GNU/Linux 7 \n \l

zalex_ua I asked Sasha and he said that he did the test on Raspberry PI 1, not 2nd gen. Regular SD card was used.

Comment by Oleksii Zagorskyi [ 2015 Aug 21 ]

What interesting detail is that such problem was discovered already 3 times (on different production installations) and in all cases for the same table "proxy_history".
ADDED: later, noted below, +2 such cases for other productions.

Errors in log files were like:

... Database is down. Retrying in 10 seconds.
... [Z3005] query failed: [0] database disk image is malformed [delete from proxy_history where id<64678908 and (clock<1355446567 or (id<=64211429 and clock<1353440509))]

This table probably gets most of writes/deletes and that's not surprised that it has more chances to become corrupted.

In one case those errors appeared in log right after proxy start (but it was able to start, including heartbeat sender !) and 2nd case - I don't know exactly, but also when proxy was running.

If you for example will create zero-byte db file, then proxy will stop immediately during start:

9112:20150821:163938.256 using configuration file: /zab/bin/2.4/zabbix_proxy.conf
9112:20150821:163938.258 [Z3005] query failed: [0] no such table: users [select userid from users limit 1] 
9112:20150821:163938.258 cannot use database "/zab/bin/2.4/zabbix_proxy2.4.db": database is not a Zabbix database

Interesting test is - the zero-byte file may appear for example if delete the db file with running proxy - then the running proxy "re-creates" the zero-byte file and keeps running (of course incorrectly and logging errors to log file).
But after restart - it will stop immediately.

Another test. When proxy is running, fill db by a garbage:

# echo "garbage" > zabbix_proxy.db

then in log:

9871:20150821:183508.804 [Z3005] query failed: [0] attempt to write a readonly database [update hosts set snmp_disable_until=1440171323,snmp_error='Timeout while connecting to "10.10.10.10:161".' where hostid=10
147]                                                                                                                                                                                                                 
9863:20150821:183519.286 received configuration data from server, datalen 3326

10069:20150821:184445.575 [Z3005] query failed: [0] file is encrypted or is not a database [select hostid,available,error,snmp_available,snmp_error,ipmi_available,ipmi_error,jmx_available,jmx_error from hosts whe
re status in (0,1) order by hostid asc]                                                                                                                                                                              
 10069:20150821:184445.576 [Z3005] query failed: [0] file is encrypted or is not a database [pragma synchronous=0]                                                                                                   
 10069:20150821:184445.577 database is down: reconnecting in 10 seconds                                                                                                                                              
 10087:20150821:184450.111 [Z3005] query failed: [0] file is encrypted or is not a database [insert into proxy_history (itemid,clock,ns,value,state) values (30657,1440171887,515867826,'Invalid second parameter.',1
);                                                                                                                                                                                                                   
]

but heartbeat message sender still keeps sending messages to server ....

When db is correct:

# ps aux | grep zabbix_proxy | grep ing
zabbix   10068  0.0  0.0 161412  3324 ?        S    18:41   0:00 ./zabbix_proxy2.4: heartbeat sender [sending heartbeat message success in 0.012326 sec, idle 60 sec]
zabbix   10076  0.0  0.0 161412  4208 ?        S    18:41   0:00 ./zabbix_proxy2.4: trapper #1 [processed data in 0.000000 sec, waiting for connection]
zabbix   10077  0.0  0.0 161412  4208 ?        S    18:41   0:00 ./zabbix_proxy2.4: trapper #2 [processed data in 0.000000 sec, waiting for connection]
zabbix   10078  0.0  0.0 161412  4184 ?        S    18:41   0:00 ./zabbix_proxy2.4: trapper #3 [processed data in 0.000000 sec, waiting for connection]
zabbix   10079  0.0  0.0 161412  4208 ?        S    18:41   0:00 ./zabbix_proxy2.4: trapper #4 [processed data in 0.000000 sec, waiting for connection]
zabbix   10080  0.0  0.0 161412  4208 ?        S    18:41   0:00 ./zabbix_proxy2.4: trapper #5 [processed data in 0.000000 sec, waiting for connection]
zabbix   10081  0.0  0.0 161756  3324 ?        S    18:41   0:00 ./zabbix_proxy2.4: icmp pinger #1 [got 0 values in 0.000006 sec, idle 5 sec]
zabbix   10089  0.0  0.0 161412  3276 ?        S    18:41   0:00 ./zabbix_proxy2.4: self-monitoring [processed data in 0.000016 sec, idle 1 sec]

When db is filled by garbage:

# ps aux | grep zabbix_proxy | grep ing
zabbix   10067  0.0  0.0 161624  6320 ?        S    18:41   0:00 ./zabbix_proxy2.4: configuration syncer [loading configuration]
zabbix   10068  0.0  0.0 161412  3324 ?        S    18:41   0:00 ./zabbix_proxy2.4: heartbeat sender [sending heartbeat message success in 0.007459 sec, idle 60 sec]
zabbix   10069  0.0  0.0 161412  5688 ?        S    18:41   0:00 ./zabbix_proxy2.4: data sender [sent 0 values in 0.001968 sec, sending data]
zabbix   10075  0.1  0.1 175372 17036 ?        S    18:41   0:00 ./zabbix_proxy2.4: unreachable poller #1 [got 0 values in 0.000005 sec, getting values]
zabbix   10076  0.0  0.0 161412  4208 ?        S    18:41   0:00 ./zabbix_proxy2.4: trapper #1 [processed data in 0.000000 sec, waiting for connection]
zabbix   10077  0.0  0.0 161412  4208 ?        S    18:41   0:00 ./zabbix_proxy2.4: trapper #2 [processed data in 0.000000 sec, waiting for connection]
zabbix   10078  0.0  0.0 161412  4184 ?        S    18:41   0:00 ./zabbix_proxy2.4: trapper #3 [processed data in 0.000000 sec, waiting for connection]
zabbix   10079  0.0  0.0 161412  4208 ?        S    18:41   0:00 ./zabbix_proxy2.4: trapper #4 [processed data in 0.000000 sec, waiting for connection]
zabbix   10080  0.0  0.0 161412  4208 ?        S    18:41   0:00 ./zabbix_proxy2.4: trapper #5 [processed data in 0.000000 sec, waiting for connection]
zabbix   10081  0.0  0.0 161756  3324 ?        S    18:41   0:00 ./zabbix_proxy2.4: icmp pinger #1 [got 0 values in 0.000005 sec, idle 5 sec]
zabbix   10083  0.0  0.0 161412  4136 ?        S    18:41   0:00 ./zabbix_proxy2.4: http poller #1 [got 0 values in 0.000701 sec, getting values]
zabbix   10084  0.1  0.1 174856 16656 ?        S    18:41   0:00 ./zabbix_proxy2.4: discoverer #1 [processed 0 rules in 0.000306 sec, performing discovery]
zabbix   10085  0.0  0.0 161412  4140 ?        S    18:41   0:00 ./zabbix_proxy2.4: history syncer #1 [synced 0 items in 0.000002 sec, syncing history]
zabbix   10087  0.0  0.0 161440  5656 ?        S    18:41   0:00 ./zabbix_proxy2.4: history syncer #3 [synced 0 items in 0.000002 sec, syncing history]
zabbix   10088  0.0  0.0 161412  4140 ?        S    18:41   0:00 ./zabbix_proxy2.4: history syncer #4 [synced 0 items in 0.000002 sec, syncing history]
zabbix   10089  0.0  0.0 161412  3276 ?        S    18:41   0:00 ./zabbix_proxy2.4: self-monitoring [processed data in 0.000004 sec, idle 1 sec]

Comment by Oleksii Zagorskyi [ 2015 Aug 24 ]

Not sure, but still ...
Taking into account previous comment if we would have such malformed db file - we could try to check what we will get for, for example:

SELECT * FROM proxy_history LIMIT 1;

of the malformed file.

Here is just a quick test:
on a zero-byte, or garbage filled db file:

# echo "SELECT * FROM proxy_history LIMIT 1;" | sqlite3 zabbix_proxy2.4.db 2>/dev/null; echo $?
1

on an fresh, correct zabbix db file:

# echo "SELECT * FROM proxy_history LIMIT 1;" | sqlite3 zabbix_proxy2.4-good.db 2>/dev/null; echo $?
0

Such test is very simple and supposedly should not take a lot of time in any case, i.e. could be integrated to init.d script if we would be sure that it's effective.

Comment by Oleksii Zagorskyi [ 2015 Aug 31 ]

I've got such corrupted db file from a production zabbix proxy (v 2.2).
It's not so big, just 9 MB size.

Some tests on it:

# echo "SELECT * FROM proxy_history LIMIT 63;" | sqlite3 zabbix_proxy.db > /dev/null
# echo $?
0

# echo "SELECT count(*) FROM proxy_history;" | sqlite3 zabbix_proxy.db > /dev/null
Error: near line 1: database disk image is malformed
# echo $?
1

I was able to select up to 63 rows limit without errors.

Proxy log file with this db file looks like:

 30737:20150831:145920.377 Starting Zabbix Proxy (active) [first_test_proxy]. Zabbix 2.2.11rc1 (revision 55263).
 30737:20150831:145920.377 **** Enabled features ****
 30737:20150831:145920.377 SNMP monitoring:       YES
 30737:20150831:145920.377 IPMI monitoring:       YES
 30737:20150831:145920.377 WEB monitoring:        YES
 30737:20150831:145920.377 VMware monitoring:     YES
 30737:20150831:145920.377 ODBC:                  YES
 30737:20150831:145920.377 SSH2 support:          YES
 30737:20150831:145920.377 IPv6 support:          YES
 30737:20150831:145920.377 **************************
 30737:20150831:145920.377 using configuration file: /zab/bin/2.2/zabbix_proxy.conf
 30737:20150831:145920.383 current database version (mandatory/optional): 02020000/02020001
 30737:20150831:145920.383 required mandatory version: 02020000
 30738:20150831:145920.389 proxy #1 started [configuration syncer #1]
 30739:20150831:145920.389 proxy #2 started [heartbeat sender #1]
 30740:20150831:145920.390 proxy #3 started [data sender #1]
 30749:20150831:145920.397 proxy #12 started [trapper #3]
 30752:20150831:145920.398 proxy #15 started [icmp pinger #1]
 30737:20150831:145920.398 proxy #0 started [main process]
 30750:20150831:145920.401 proxy #13 started [trapper #4]
 30754:20150831:145920.402 proxy #17 started [http poller #1]
 30747:20150831:145920.402 proxy #10 started [trapper #1]
 30751:20150831:145920.402 proxy #14 started [trapper #5]
 30756:20150831:145920.403 proxy #19 started [history syncer #1]
 30760:20150831:145920.404 proxy #23 started [self-monitoring #1]
 30753:20150831:145920.407 proxy #16 started [housekeeper #1]
 30753:20150831:145920.407 executing housekeeper
 30757:20150831:145920.411 proxy #20 started [history syncer #2]
 30759:20150831:145920.411 proxy #22 started [history syncer #4]
 30748:20150831:145920.422 proxy #11 started [trapper #2]
 30758:20150831:145920.426 proxy #21 started [history syncer #3]
 30738:20150831:145920.429 received configuration data from server, datalen 3201
 30753:20150831:145920.469 [Z3005] query failed: [0] database disk image is malformed [select max(id) from proxy_history]
 30744:20150831:145920.924 proxy #7 started [poller #4]
 30746:20150831:145920.952 proxy #9 started [unreachable poller #1]
 30743:20150831:145920.987 proxy #6 started [poller #3]
 30742:20150831:145920.990 proxy #5 started [poller #2]
 30755:20150831:145921.017 proxy #18 started [discoverer #1]
 30745:20150831:145921.019 proxy #8 started [poller #5]
 30741:20150831:145921.025 proxy #4 started [poller #1]

note above a failed query "select max(id) from proxy_history".
Zabbix proxy, obviously, could not start to operate normally, but the "heartbeat" is working:

# ps aux | grep zabbix_proxy | grep ing
zabbix   30738  0.0  0.0 161636  5684 ?        S    14:59   0:00 ./zabbix_proxy2.2: configuration syncer [loading configuration]
zabbix   30739  0.0  0.0 161424  3308 ?        S    14:59   0:00 ./zabbix_proxy2.2: heartbeat sender [sending heartbeat message success in 0.017676 sec, idle 60 sec]
zabbix   30740  0.0  0.0 161424  4144 ?        S    14:59   0:00 ./zabbix_proxy2.2: data sender [sent 0 values in 0.000000 sec, sending data]
zabbix   30741  0.2  0.1 174412 15408 ?        S    14:59   0:00 ./zabbix_proxy2.2: poller #1 [connecting to the database]
zabbix   30742  0.2  0.1 174412 15408 ?        S    14:59   0:00 ./zabbix_proxy2.2: poller #2 [connecting to the database]
zabbix   30743  0.2  0.1 174412 15408 ?        S    14:59   0:00 ./zabbix_proxy2.2: poller #3 [connecting to the database]
zabbix   30744  0.2  0.1 174412 15408 ?        S    14:59   0:00 ./zabbix_proxy2.2: poller #4 [connecting to the database]
zabbix   30745  0.2  0.1 174412 15408 ?        S    14:59   0:00 ./zabbix_proxy2.2: poller #5 [connecting to the database]
zabbix   30746  0.2  0.1 174412 15408 ?        S    14:59   0:00 ./zabbix_proxy2.2: unreachable poller #1 [connecting to the database]
zabbix   30747  0.0  0.0 161424  4204 ?        S    14:59   0:00 ./zabbix_proxy2.2: trapper #1 [processed data in 0.000000 sec, waiting for connection]
zabbix   30748  0.0  0.0 161424  4204 ?        S    14:59   0:00 ./zabbix_proxy2.2: trapper #2 [processed data in 0.000000 sec, waiting for connection]
zabbix   30749  0.0  0.0 161424  4204 ?        S    14:59   0:00 ./zabbix_proxy2.2: trapper #3 [processed data in 0.000000 sec, waiting for connection]
zabbix   30750  0.0  0.0 161424  4204 ?        S    14:59   0:00 ./zabbix_proxy2.2: trapper #4 [processed data in 0.000000 sec, waiting for connection]
zabbix   30751  0.0  0.0 161424  4204 ?        S    14:59   0:00 ./zabbix_proxy2.2: trapper #5 [processed data in 0.000000 sec, waiting for connection]
zabbix   30752  0.0  0.0 161744  3304 ?        S    14:59   0:00 ./zabbix_proxy2.2: icmp pinger #1 [got 0 values in 0.000005 sec, idle 5 sec]
zabbix   30753  0.0  0.0 161424  4332 ?        S    14:59   0:00 ./zabbix_proxy2.2: housekeeper [removing old history]
zabbix   30754  0.0  0.0 161424  4140 ?        S    14:59   0:00 ./zabbix_proxy2.2: http poller #1 [got 0 values in 0.000000 sec, getting values]
zabbix   30755  0.2  0.1 174412 15408 ?        S    14:59   0:00 ./zabbix_proxy2.2: discoverer #1 [connecting to the database]
zabbix   30760  0.0  0.0 161424  3308 ?        S    14:59   0:00 ./zabbix_proxy2.2: self-monitoring [processed data in 0.000007 sec, idle 1 sec]

So according this case, we would need to scan the table completely to make sure it's readable.
At least it should be (supposedly) faster than "pragma integrity_check;" and could be (theoretically) acceptable as for init script.

Let's check by "pragma integrity_check":

# time echo 'pragma integrity_check;' | sqlite3 zabbix_proxy.db 
*** in database main ***
Fragmentation of 352 bytes reported as 0 on page 8885
On tree page 8873 cell 57: Extends off end of page
On tree page 353 cell 63: Rowid 11 out of order
On tree page 8850 cell 63: Extends off end of page
On tree page 353 cell 62: Rowid 11268148 out of order
On tree page 7749 cell 6: Rowid 11565396 out of order
On tree page 353 cell 60: Child page depth differs
On tree page 7219 cell 7: Rowid 11567553 out of order
On tree page 7081 cell 7: Rowid 11569566 out of order
On tree page 7218 cell 6: Rowid 11572396 out of order
On tree page 7135 cell 9: Rowid 11569666 out of order
On tree page 7163 cell 8: Rowid 11569310 out of order
On tree page 7221 cell 11: Rowid 11569536 out of order
On tree page 7211 cell 6: Rowid 11569367 out of order
On tree page 7217 cell 6: Rowid 11569446 out of order
On tree page 353 cell 58: Child page depth differs
On tree page 7249 cell 8: Rowid 11569516 out of order
On tree page 7147 cell 7: Rowid 11569500 out of order
On tree page 7128 cell 8: Rowid 11569582 out of order
On tree page 7122 cell 6: Rowid 11571233 out of order
On tree page 353 cell 57: Child page depth differs
On tree page 353 cell 56: Child page depth differs
On tree page 353 cell 55: Child page depth differs
Page 7245: btreeInitPage() returns error code 11
On tree page 353 cell 54: Child page depth differs
Page 7381: btreeInitPage() returns error code 11
Page 7082: btreeInitPage() returns error code 11
On tree page 353 cell 52: Child page depth differs
On tree page 7330 cell 6: Rowid 11551050 out of order
On tree page 353 cell 51: Child page depth differs
Page 8879: btreeInitPage() returns error code 11
Page 8878: btreeInitPage() returns error code 11
On tree page 8877 cell 21: 2nd reference to page 546
On tree page 8877 cell 88: Rowid 11524755 out of order
On tree page 8877 cell 88: 2nd reference to page 1300
On tree page 8877 cell 87: 2nd reference to page 5746
On tree page 8877 cell 86: 2nd reference to page 5519
On tree page 8877 cell 85: 2nd reference to page 1101
On tree page 8877 cell 84: 2nd reference to page 2392
On tree page 8877 cell 83: 2nd reference to page 1450
On tree page 8877 cell 82: 2nd reference to page 5299
On tree page 8877 cell 81: 2nd reference to page 4654
On tree page 8877 cell 80: 2nd reference to page 4740
On tree page 8877 cell 79: 2nd reference to page 2981
On tree page 8877 cell 78: 2nd reference to page 725
On tree page 8877 cell 77: 2nd reference to page 1625
On tree page 8877 cell 76: 2nd reference to page 5415
On tree page 8877 cell 75: 2nd reference to page 3401
On tree page 8877 cell 74: 2nd reference to page 1790
On tree page 8877 cell 73: 2nd reference to page 7068
On tree page 8877 cell 72: 2nd reference to page 4253
On tree page 8877 cell 71: 2nd reference to page 2432
On tree page 8877 cell 70: 2nd reference to page 1059
On tree page 8877 cell 69: 2nd reference to page 3466
On tree page 8877 cell 68: 2nd reference to page 4501
On tree page 8877 cell 67: 2nd reference to page 6212
On tree page 8877 cell 66: 2nd reference to page 1700
On tree page 8877 cell 65: 2nd reference to page 2782
On tree page 8877 cell 64: 2nd reference to page 6729
On tree page 8877 cell 63: 2nd reference to page 5414
On tree page 8877 cell 62: 2nd reference to page 2192
On tree page 8877 cell 61: 2nd reference to page 6384
On tree page 8877 cell 60: 2nd reference to page 6576
On tree page 8877 cell 59: 2nd reference to page 850
On tree page 8877 cell 58: 2nd reference to page 2530
On tree page 8877 cell 57: 2nd reference to page 745
On tree page 8877 cell 56: 2nd reference to page 6440
On tree page 8877 cell 55: 2nd reference to page 837
On tree page 8877 cell 54: 2nd reference to page 1367
On tree page 8877 cell 53: 2nd reference to page 1724
On tree page 8877 cell 52: 2nd reference to page 6743
On tree page 8877 cell 51: 2nd reference to page 3566
On tree page 8877 cell 50: 2nd reference to page 2372
On tree page 8877 cell 49: 2nd reference to page 6076
On tree page 8877 cell 48: 2nd reference to page 2484
On tree page 8877 cell 47: 2nd reference to page 6342
On tree page 8877 cell 46: 2nd reference to page 1580
On tree page 8877 cell 45: 2nd reference to page 6455
On tree page 8877 cell 44: 2nd reference to page 1221
On tree page 8877 cell 43: 2nd reference to page 7699
On tree page 8877 cell 42: 2nd reference to page 1961
On tree page 8877 cell 41: 2nd reference to page 3101
On tree page 8877 cell 40: 2nd reference to page 6621
On tree page 8877 cell 39: 2nd reference to page 3492
On tree page 8877 cell 38: 2nd reference to page 612
On tree page 8877 cell 37: 2nd reference to page 6964
On tree page 8877 cell 36: 2nd reference to page 492
On tree page 8877 cell 35: 2nd reference to page 7377
On tree page 8877 cell 34: 2nd reference to page 1965
On tree page 8877 cell 33: 2nd reference to page 3658
On tree page 8877 cell 32: 2nd reference to page 6011
On tree page 8877 cell 31: 2nd reference to page 512
On tree page 8877 cell 30: 2nd reference to page 858
On tree page 8877 cell 29: 2nd reference to page 6584
On tree page 8877 cell 28: 2nd reference to page 7045
On tree page 8877 cell 27: 2nd reference to page 7262
On tree page 8877 cell 26: 2nd reference to page 5925
On tree page 8877 cell 25: 2nd reference to page 4554
On tree page 8877 cell 24: 2nd reference to page 1675
On tree page 8877 cell 23: 2nd reference to page 5705

real    0m0.013s
user    0m0.004s
sys     0m0.004s

# echo 'pragma integrity_check;' | sqlite3 zabbix_proxy.db > /dev/null
# echo $?
0

note that the exit code is 0, so in initscripts a STDOUT should be checked for a single-line "ok" text only.

On a clean DB file we get the same exit code 0:

# echo 'pragma integrity_check;' | sqlite3 zabbix_proxy_clean.db
ok
# echo $?
0

Conclusion, IMO:
In init scripts I'd go with echo "SELECT count (*) FROM proxy_history;" | sqlite3 zabbix_proxy.db &> /dev/null and checking exitcode.
If the exitcode is 1 - then delete the db file.

hmm, interesting thing is exitcode when getting the result in different ways, be careful:

# echo "SELECT count(*) FROM proxy_history;" | sqlite3 zabbix_proxy.db &> /dev/null; echo $?
1
# sqlite3 zabbix_proxy.db "SELECT count(*) FROM proxy_history;" &> /dev/null; echo $?
11

# echo "SELECT count(*) FROM proxy_history;" | sqlite3 zabbix_proxy.db; echo $?
Error: near line 1: database disk image is malformed
1
# sqlite3 zabbix_proxy.db "SELECT count(*) FROM proxy_history;"; echo $?
Error: database disk image is malformed
11

and with manually corrupted db:

# echo "garbage" > garbage.db
# echo "SELECT count(*) FROM proxy_history;" | sqlite3 garbage.db; echo $?
Error: near line 1: file is encrypted or is not a database
1
# sqlite3 garbage.db "SELECT count(*) FROM proxy_history;"; echo $?
Error: file is encrypted or is not a database
26

# echo "" > garbage.db
# echo "SELECT count(*) FROM proxy_history;" | sqlite3 garbage.db; echo $?
Error: near line 1: no such table: proxy_history
1
# sqlite3 garbage.db "SELECT count(*) FROM proxy_history;"; echo $?
Error: no such table: proxy_history
1

Comment by Oleksii Zagorskyi [ 2016 Mar 22 ]

Another corrupted database case from production:

# sqlite3 sqlite.zabbix_proxy.db "SELECT count(*) FROM proxy_history;"; echo $?
Error: database disk image is malformed
11

# echo "SELECT count(*) FROM proxy_history;" | sqlite3 sqlite.zabbix_proxy.db; echo $?
Error: near line 1: database disk image is malformed
1

Comment by Oleksii Zagorskyi [ 2017 Jan 14 ]

One more case from a production:

24638:20170104:010930.002 [Z3005] query failed: [0] database disk image is malformed [delete from proxy_history where id<2913563192 and (clock<1483319366 or (id<=2913556555 and clock<1483492166))]
24485:20170104:033953.371 [Z3005] query failed: [0] database disk image is malformed [insert into proxy_history (itemid,clock,ns,value) values (891319,1483490199,910633193,'xxx');
12102:20170111:181524.589 [Z3005] query failed: [0] database disk image is malformed [delete from proxy_history where id<2913563192 and (clock<1483985722 or (id<=2913556555 and clock<1483553957))]

looks like happened during the proxy operation, but again, with the "proxy_history" table.
If during proxy (re)start and checking the table we would remove the db file automatically - that would be user friendly enough.

Comment by Tony den Haan [ 2023 Apr 07 ]

I spotted "query failed: [0] database disk image is malformed" after which proxy happily went on burning electrons. Shouldn't it log and then quit on such critical error?

[ZBX-6036] Zabbix proxy can start if DB is down or corrupted Created: 2012 Dec 28 Updated: 2023 Apr 07 Resolved: 2015 Mar 26
Status:	Closed
Project:	ZABBIX BUGS AND ISSUES
Component/s:	Proxy (P)
Affects Version/s:	1.8.15, 2.0.4
Fix Version/s:	None

[ZBX-6036] Zabbix proxy can start if DB is down or corrupted Created: 2012 Dec 28 Updated: 2023 Apr 07 Resolved: 2015 Mar 26