-
Incident report
-
Resolution: Commercial support required
-
Trivial
-
None
-
7.0.4
-
None
-
zabbix cluster with separate servers/VMs for the database, the server, the frontend, end for proxies, all running on Rocky9 with mariadb 10.11
Upgrade from 6.4 to 7.0 did not go well, so I started over with all fresh and new 7.0.4 server.
I have a fresh and clean Zabbix 7 installation, and every tome I add an snmp monitored host with a proxy the db gets corrupted.
It took me several start-overs to find this reprocducible.
Being careful by adding hosts I managed to get to 53 hosts now, and when adding my IPFire it corrupted the DB again (although this time without putting any proxy (but with proxy it is reproducible).
Steps to reproduce:
- Add a host monitored by snmp with a proxy (e.g. if cloning another already integrated switch)
- It just takes a few second to 2 minutes and the database gets corrupted, with no monitring data anymore in the GUI.
Result:
See screenshot attached in dashboard. Same is visible in Monitoring -> Hosts. System is useles now.
In the server log the corresponding entry is
5396:20241015:162444.654 [Z3005] query failed: [126] Table zabbix/history_uint in tablespace 140636221125656 corrupted. [insert into history_uint (itemid,clock,ns,value) values (48583,1729027483,779428830,1),(48685,1729027481,84194830,1),(48703,1729027483,779338891,1),(50505,1729027483,594920014,2),(50508,1729027482,981465793,3),(50511,1729027482,981465793,3),(50512,1729027482,981465793,3),(50513,1729027482,981465793,3),(50514,1729027482,981465793,3),(50515,1729027482,981465793,3),(50518,1729027482,981465793,3),(50526,1729027482,981465793,3),(50530,1729027482,981465793,3),(50531,1729027482,981465793,3),(50532,1729027482,981465793,3),(50533,1729027482,981465793,3),(50535,1729027482,981465793,3),(50537,1729027482,981465793,3),(50538,1729027482,981465793,3),(50542,1729027482,981465793,3),(50544,1729027482,981465793,3),(50548,1729027482,981465793,3),(50553,1729027482,981465793,3),(50555,1729027482,981465793,3),(50558,1729027482,981465793,3),(50559,1729027482,981465793,3),(50562,1729027483,291373300,0),(50564,1729027483,594920014,0),(50565,1729027483,594920014,0),(50566,1729027483,594920014,0),(50567,1729027483,291373300,0),(50568,1729027483,594920014,0),(50569,1729027483,291373300,0),(50573,1729027483,594920014,0),(50577,1729027483,291373300,0),(50579,1729027482,981465793,0),(50580,1729027483,594920014,0),(50582,1729027483,291373300,0),(50591,1729027483,291373300,0),(50595,1729027483,291373300,0),(50597,1729027483,594920014,0),(50598,1729027483,291373300,0),(50600,1729027483,594920014,0),(50602,1729027483,594920014,0),(50608,1729027483,594920014,0),(50609,1729027483,594920014,5),(50610,1729027483,291373300,0),(50611,1729027483,291373300,0),(50615,1729027483,594920014,0),(50617,1729027483,291373300,0),(50620,1729027482,981465793,0),(50622,1729027483,291373300,0),(50624,1729027483,594920014,0),(50625,1729027483,594920014,0),(50627,1729027483,594920014,0),(50629,1729027483,291373300,0),(50631,1729027483,291373300,0),(50632,1729027483,291373300,0),(50635,1729027482,981465793,0),(50643,1729027483,291373300,0),(50644,1729027483,594920014,0),(50646,1729027483,291373300,0),(50652,1729027483,291373300,0),(50656,1729027482,981465793,0),(50661,1729027482,981465793,0),(50664,1729027483,291373300,0),(50671,1729027483,291373300,0),(50681,1729027482,981465793,0),(50682,1729027482,981465793,0),(50684,1729027483,291373300,0),(50691,1729027483,291373300,0),(50700,1729027483,291373300,0),(50705,1729027483,291373300,0),(50706,1729027482,981465793,0),(50710,1729027483,594920014,85392),(50713,1729027483,291373300,1840),(50715,1729027483,594920014,1848),(50722,1729027483,594920014,0),(50727,1729027482,981465793,240),(50729,1729027483,594920014,16),(50730,1729027483,594920014,119168),(50735,1729027482,981465793,0),(50736,1729027483,291373300,128),(50744,1729027483,291373300,0),(50747,1729027483,594920014,1480),(50748,1729027483,594920014,1480),(50764,1729027482,981465793,0),(50769,1729027483,291373300,0),(50781,1729027482,981465793,0),(50786,1729027483,291373300,0),(50787,1729027483,594920014,0),(50791,1729027483,594920014,0),(50792,1729027483,291373300,0),(50793,1729027482,981465793,0),(50794,1729027482,981465793,0),(50796,1729027482,981465793,0),(50806,1729027482,981465793,0),(50808,1729027483,594920014,0),(50809,1729027483,594920014,0),(50810,1729027483,594920014,0),(50814,1729027483,594920014,0),(50816,1729027483,291373300,0),(50818,1729027482,981465793,0),(50823,1729027483,594920014,0),(50827,1729027482,981465793,0),(50828,1729027483,594920014,0),(50831,1729027483,594920014,0),(50832,1729027483,594920014,0),(50835,1729027483,291373300,0),(50839,1729027483,594920014,0),(50844,1729027483,291373300,0),(50845,1729027483,594920014,0),(50850,1729027483,594920014,0),(50851,1729027483,594920014,0),(50857,1729027483,594920014,0),(50860,1729027483,594920014,0),(50867,1729027483,594920014,0),(50874,1729027483,291373300,0),(50885,1729027483,291373300,0),(50887,1729027483,594920014,0),(50888,1729027482,981465793,0),(50889,1729027483,594920014,0),(50892,1729027483,291373300,0),(50894,1729027482,981465793,0),(50895,1729027483,291373300,0),(50904,1729027483,291373300,0),(50909,1729027483,291373300,0),(50915,1729027483,594920014,0),(50916,1729027483,291373300,0),(50918,1729027483,291373300,0),(50922,1729027483,291373300,0),(50935,1729027483,594920014,17256),(50938,1729027483,594920014,1008),(50943,1729027483,291373300,26288),(50951,1729027483,291373300,144200),(50957,1729027482,981465793,119048),(50958,1729027482,981465793,22496),(50961,1729027483,594920014,1736),(50969,1729027483,291373300,3200),(50991,1729027483,594920014,0),(50995,1729027483,594920014,0),(50997,1729027483,594920014,0),(51076,1729027482,981465793,1),(51080,1729027482,981465793,1),(51081,1729027482,981465793,1),(51082,1729027482,981465793,1),(51086,1729027482,981465793,1),(51087,1729027482,981465793,1),(51090,1729027482,981465793,1),(51092,1729027482,981465793,1),(51094,1729027482,981465793,1),(51099,1729027482,981465793,1),(51101,1729027482,981465793,1),(51104,1729027482,981465793,1),(51107,1729027482,981465793,1),(51108,1729027482,981465793,1),(51109,1729027482,981465793,1),(51116,1729027482,981465793,1),(51118,1729027482,981465793,1),(51125,1729027482,981465793,1),(51126,1729027482,981465793,1),(51137,1729027482,981465793,1),(51138,1729027482,981465793,1),(51140,1729027483,594920014,1),(51141,1729027482,981465793,2),(51239,1729027483,507548307,4381),(51252,1729027483,507548307,52),(51262,1729027483,507548307,0),(51267,1729027483,507548307,0),(51271,1729027483,507548307,0),(51281,1729027483,507548307,0),(51284,1729027483,507548307,0),(51291,1729027483,507548307,0),(51292,1729027483,507548307,0),(51309,1729027483,507548307,0),(51313,1729027483,507548307,0),(51315,1729027483,507548307,0),(51320,1729027483,507548307,0),(51324,1729027483,507548307,0),(51327,1729027483,507548307,0),(51336,1729027483,507548307,0),(51350,1729027483,507548307,0),(51358,1729027483,507548307,0),(51370,1729027483,507548307,0),(51373,1729027483,507548307,0),(51374,1729027483,507548307,0),(51389,1729027483,507548307,0),(51401,1729027483,507548307,88),(51420,1729027483,507548307,5910448),(51426,1729027483,507548307,1424),(51429,1729027483,507548307,5050536),(51442,1729027483,507548307,0),(51459,1729027483,507548307,12250112),(51474,1729027483,507548307,0),(51486,1729027483,507548307,0),(51493,1729027483,507548307,0),(51500,1729027483,507548307,0),(51502,1729027483,507548307,0),(51505,1729027483,507548307,0),(51518,1729027483,507548307,0),(51531,1729027483,507548307,0),(51567,1729027483,507548307,0),(51570,1729027483,507548307,0),(51610,1729027483,507548307,0),(51613,1729027483,507548307,0),(51644,1729027483,507548307,0),(51653,1729027483,507548307,0),(51667,1729027483,507548307,22136),(51702,1729027483,507548307,19280),(51725,1729027483,507548307,0),(51749,1729027483,507548307,0),(51843,1729027483,507548307,1),(51864,1729027483,507548307,1),(51869,1729027483,507548307,1),(51874,1729027483,507548307,1),(51881,1729027483,507548307,1),(51887,1729027483,507548307,2),(51893,1729027483,507548307,1),(51901,1729027483,507548307,1),(51903,1729027483,507548307,2),(51910,1729027483,507548307,2),(51929,1729027483,507548307,1),(52032,1729027483,507548307,3),(52044,1729027483,507548307,3),(52051,1729027483,507548307,3),(52054,1729027483,507548307,3),(52075,1729027483,507548307,3),(55714,1729027483,803368677,80),(55715,1729027483,803368677,2840),(55716,1729027483,803368677,12400),(55718,1729027483,803368677,0),(55721,1729027483,803368677,0),(55722,1729027483,803368677,0),(55723,1729027483,803368677,1528),(55724,1729027483,803368677,0),(55725,1729027483,803368677,128),(55726,1729027483,803368677,128),(55876,1729027483,803368677,864),(55878,1729027483,803368677,58424),(55879,1729027483,803368677,58424),(55886,1729027483,803368677,0),(55887,1729027483,803368677,0),(55888,1729027483,803368677,1648),(55889,1729027483,803368677,0),(55890,1729027483,803368677,0),(55891,1729027483,803368677,0),(55892,1729027483,803368677,0),(55893,1729027483,803368677,0),(55894,1729027483,803368677,0),(55895,1729027483,803368677,0),(55896,1729027483,803368677,0),(55897,1729027483,803368677,0),(55898,1729027483,803368677,5747416),(55899,1729027483,803368677,0),(55900,1729027483,803368677,5544),(55901,1729027483,803368677,0),(55902,1729027483,803368677,904),(55903,1729027483,803368677,0),(55904,1729027483,803368677,0),(55905,1729027483,803368677,11424),(55906,1729027483,803368677,248),(55907,1729027483,803368677,0),(55908,1729027483,803368677,0),(55909,1729027483,803368677,2168),(55910,1729027483,803368677,880),(55911,1729027483,803368677,880),(55912,1729027483,803368677,0),(55913,1729027483,803368677,0),(55914,1729027483,803368677,864),(55915,1729027483,803368677,904),(55916,1729027483,803368677,0),(55917,1729027483,803368677,0),(55918,1729027483,803368677,2168),(55919,1729027483,803368677,0),(55920,1729027483,803368677,0),(55921,1729027483,803368677,0),(55922,1729027483,803368677,480),(55923,1729027483,803368677,0),(55924,1729027483,803368677,0),(55925,1729027483,803368677,5056),(55979,1729027483,803368677,1),(55980,1729027483,803368677,1),(55981,1729027483,803368677,1),(55982,1729027483,803368677,2),(55983,1729027483,803368677,1),(55984,1729027483,803368677,1),(55985,1729027483,803368677,1),(55986,1729027483,803368677,1),(55987,1729027483,803368677,1),(55988,1729027483,803368677,1),(55989,1729027483,803368677,2),(55990,1729027483,803368677,1),(55991,1729027483,803368677,1),(55992,1729027483,803368677,2),(55993,1729027483,803368677,1),(55994,1729027483,803368677,1),(55995,1729027483,803368677,1),(55996,1729027483,803368677,1),(55997,1729027483,803368677,2),(55998,1729027483,803368677,1),(55999,1729027483,803368677,1),(56000,1729027483,803368677,1),(56001,1729027483,803368677,2),(56002,1729027483,803368677,2),(56003,1729027483,803368677,1),(56004,1729027483,803368677,1),(56005,1729027483,803368677,2),(56006,1729027483,803368677,1),(56007,1729027483,803368677,2),(56008,1729027483,803368677,1),(56009,1729027483,803368677,2),(56010,1729027483,803368677,1),(56011,1729027483,803368677,1),(56012,1729027483,803368677,1),(56013,1729027483,803368677,1),(56014,1729027483,803368677,1),(56015,1729027483,803368677,1),(56016,1729027483,803368677,1),(56017,1729027483,803368677,2),(56018,1729027483,803368677,1),(56019,1729027483,803368677,2),(56020,1729027483,803368677,1),(56021,1729027483,803368677,1),(56022,1729027483,803368677,1),(56023,1729027483,803368677,2),(56024,1729027483,803368677,1),(56025,1729027483,803368677,1),(56026,1729027483,803368677,2),(56027,1729027483,803368677,2),(56028,1729027483,803368677,1),(56029,1729027483,803368677,1),(56030,1729027483,803368677,1),(56031,1729027483,803368677,1);
This fills up the log basically every couple second.
