systemctl.png
zabbix_server.log
[ZBX-16361] Zabbix Server stopped. Zabbix 4.2.1 (revision 92832). Created: 2019 Jul 10 Updated: 2019 Jul 24 Resolved: 2019 Jul 11 |
|
| Status: | Closed |
| Project: | ZABBIX BUGS AND ISSUES |
| Component/s: | Server (S) |
| Affects Version/s: | 4.2.3 |
| Fix Version/s: | None |
| Type: | Incident report | Priority: | Critical |
| Reporter: | Robbert van der Kooi | Assignee: | DaneT |
| Resolution: | Fixed | Votes: | 0 |
| Labels: | crash | ||
| Remaining Estimate: | Not Specified | ||
| Time Spent: | Not Specified | ||
| Original Estimate: | Not Specified | ||
| Environment: |
Ubuntu 16.04 |
||
| Attachments: |
systemctl.png
zabbix_server.log
|
| Description |
|
Hello, Todat we get a problem in the zabbix server that was: More than 95% used in the value cache I think i set the zabbix server in Maintenance mode en reboot the server. after reboot the server Zabbix won't start now. 29514:20190710:103915.638 syncing history data... 29514:20190710:103915.782 syncing history data... 71.186441% 29514:20190710:103915.784 syncing history data... 72.881356% 29514:20190710:103915.785 syncing history data... 74.576271% 29514:20190710:103915.786 syncing history data... 76.271186% 29514:20190710:103915.788 syncing history data... 77.966102% 29514:20190710:103915.790 syncing history data... 79.661017% 29514:20190710:103915.791 syncing history data... 81.355932% 29514:20190710:103915.793 syncing history data... 83.050847% 29514:20190710:103915.805 syncing history data... 100.000000% 29514:20190710:103915.805 syncing history data done 29514:20190710:103915.805 syncing trend data... 29514:20190710:103920.272 slow query: 4.410010 sec, "insert into trends_uint (itemid,clock,num,value_min,value_avg,value_max) values (77415,1562745600,39,1,1,1),(58001,1562745600,8,0, 0,0),(60954,1562745600,41,0,0,0),(41834,1562745600 ,39,0,0,0),(36299,1562745600,39,0,4621,141312),(75 865,1562745600,39,0,0,0),(73301,1562745600,39,0,1, 8),(47562,1562745600,39,0,0,0),(66290,1562745600,1 2,0,0,0),(60446,1562745600,9,0,0,0),(69887,1562745 600,40,0,0,0),(36819,1562745600,39,0,0,0),(39743,1 562745600,39,23,36,53),(65001,1562745600,12,0,0,0) ,(57405,1562745600,13,10504,22477,105336),(61076,1 562745600,41,0,0,0),(62400,1562745600,41,0,0,0),(7 1546,1562745600,39,0,0,0),(67733,1562745600,8,0,0, 0),(67124,1562745600,38,221080,356480,736376),(489 60,1562745600,4,2,2,2),(36697,1562745600,39,256867 203127,256867203127,256867203127),(58991,156274560 0,8,0,0,0),(46783,1562745600,39,0,0,0),(72232,1562 745600,39,0,0,0),(47295,1562745600,39,0,0,0),(3023 6,1562745600,39,0,0,0),(62344,1562745600,40,124736 ,8358561,52966760),(76000,1562745600,39,4854906880 ,4854906880,4854906880),(44966,1562745600,39,0,0,0 ),(77934,1562745600,40,8938364928,8938374758,89383 85408),(73204,1562745600,40,255,255,255),(44403,15 62745600,40,6,6,6),(72292,1562745600,39,255,255,25 5),(39726,1562745600,40,1,1,1),(71753,1562745600,3 9,255,255,255),(61522,1562745600,42,0,0,0),(37442, 1562745600,40,0,8217,68608),(40314,1562745600,39,2 176950272,2176950272,2176950272),(37062,1562745600 ,39,2175795200,2209000106,2298478592),(43640,15627 45600,39,0,0,0),(74945,1562745600,13,0,0,0),(58041 ,1562745600,13,0,0,0),(43392,1562745600,40,6,6,6), (35166,1562745600,30,0,0,0),(76645,1562745600,39,9 2833005568,92833005568,92833005568),(75931,1562745 600,39,0,0,0),(61473,1562745600,41,0,0,0),(73107,1 562745600,14,240,251,264),(56943,1562745600,8,0,0, 0),(75551,1562745600,40,0,0,0),(74121,1562745600,8 ,0,0,0),(49999,1562745600,39,0,0,0),(46654,1562745 600,39,0,0,0),(76779,1562745600,39,0,0,0),(68588,1 562745600,8,0,0,0),(74867,1562745600,8,0,0,0),(739 93,1562745600,40,1,1,1),(65652,1562745600,8,0,0,0) ,(29990,1562745600,39,66,66,67),(65136,1562745600, 8,0,0,0),(41528,1562745600,39,0,0,0),(43195,156274 5600,39,0,0,0),(61769,1562745600,13,0,0,0),(76854, 1562745600,39,0,0,0),(74023,1562745600,39,58325086 208,58367987186,58410545152),(65776,1562745600,12, 0,0,0),(46220,1562745600,39,0,0,0),(69229,15627456 00,39,0,0,0),(30203,1562745600,39,0,0,0),(56657,15 62745600,39,255,255,255),(51188,1562745600,8,0,0,0 ),(69225,1562745600,39,0,0,0),0,12,0,0,0),(62733,1 562745600,40,0,0,0),(69879,1562745600,40,0,0,0),(6 6144,1562745600,8,0,0,0),(60408,1562745600,9,0,0,0 ),(61899,1562745600,13,5160,8775,14696),(77514,156 2745600,39,0,262,5120),(69294,1562745600,39,4472,4 4308,325264),(68606,1562745600,39,0,0,0),(59635,15 62745600,14,1240,14795,71024),(36298,1562745600,39 ,72,178,328),(72633,1562745600,39,0,0,0),(65497,15 62745600,8,0,0,0),(65939,1562745600,8,0,0,0),(5687 0,1562745600,8,0,0,0),(69921,1562745600,40,0,0,0), (31294,1562745600,39,0,0,0),(45053,1562745600,39,4 5840,50157,60944),(46366,1562745600,39,0,0,0),(597 91,1562745600,9,0,0,0),(38139,1562745600,39,903662 93775,90366293775,90366293775),(45706,1562745600,3 9,946,965,1008),(74009,1562745600,39,11264,47734,3 86048),(59835,1562745600,14,359824,806912,1834112) ,(45321,1562745600,1,85372956672,85372956672,85372 956672),(46391,1562745600,40,0,0,0),(74954,1562745 600,13,0,0,0),(45294,1562745600,1,1786703872,17867 03872,1786703872),(30231,1562745600,39,0,0,0),((65 512,1562745600,8,0,0,0),(69908,1562745600,40,0,0,0 ),(65030,1562745600,8,0,0,0),(46350,1562745600,39, 0,0,0),(30215,1562745600,39,0,0,0),(38563,15627456 00,39,69,132,179),(38222,1562745600,40,4,4,4),(392 17,1562745600,39,4,4,4),(75698,1562745600,39,0,0,0 ),(29672,1562745600,39,0,0,3),(51266,1562745600,8, 0,0,0),(43727,1562745600,78,2,2,2),(48778,15627456 00,78,2,2,2),(70479,1562745600,79,4,4,4),(74800,15 62745600,13,0,0,0),(77230,1562745600,40,5413844425 32,541384442532,541384442532),(76673,1562745600,39 ,0,78,1024),(46742,1562745600,40,0,0,0),(75597,156 2745600,39,0,0,0),(66273,1562745600,12,2544,3020,3 424),(67812,1562745600,8,0,0,0),(61138,1562745600, 41,0,0,0),(77022,1562745600,39,0,0,0),(71617,15627 45600,39,0,0,0),(61939,1562745600,13,0,0,0),(73923 ,1562745600,40,45295123425,45295123425,45295123425 ),(65938,1562745600,8,0,0,0),(38173,1562745600,40, 90366293775,90366293775,90366293775),(59028,156274 5600,8,0,0,0),(61524,1562745600,42,0,0,0),(32264,1 562745600,8,0,0,0),(45947,1562745600,39,0,0,0),(30 218,1562745600,39,0,0,0),(63699,1562745600,8,0,0,0 ),(74135,1562745600,12,6630864,13405722,29708768), (76697,1562745600,39,0,78,2048),(63710,1562745600, 8,0,0,0),(61930,1562745600,13,0,0,0),(47698,156274 5600,39,0,0,0),(64896,1562745600,8,0,0,0),(35204,1 562745600,3,2,2,2),(41510,1562745600,39,0,0,0),(58 721,1562745600,39,0,0,0),(61858,1562745600,8,0,0,0 ),(76487,1562745600,39,4138729472,4153167556,41597 00992),(46186,1562745600,39,5296,14599,81568),(355 39,1562745600,39,0,0,0),(43232,1562745600,39,0,0,0 ),(59160,1562745600,8,0,0,0),(73101,1562745600,8,0 ,0,0),(75446,1562745600,39,1,1,1),(57150,156274560 0,8,0,0,0),(54679,1562745600,7,0,0,0),(64967,15627 45600,12,120,182,232),(31299,1562745600,39,0,0,0), (63707,1562745600,8,0,0,0),(65577,1562745600,8,0,0 ,0),(69797,1562745600,38,0,0,0),(47608,1562745600, 1,85530243072,85530243072,85530243072),(51006,1562 745600,74,2,2,2),(75712,1562745600,39,0,0,0),(7166 6,1562745600,39,0,0,0),(59761,1562745600,9,0,0,0), (38224,1562745600,40,36000000,46325000,103000000), (34816,1562745600,39,0,0,0),(59532,1562745600,9,0, 0,0),(76965,1562745600,39,0,0,0),(47354,1562745600 ,40,6,6,6),(63137,1562745600,235,1,1,1),(56748,156 2745600,80,1,1,1),(36547,1562745600,40,0,0,0),(660 28,1562745600,8,0,0,0),(51181,1562745600,8,0,0,0), (60488,1562745600,14,0,0,0),(71637,1562745600,39,0 ,0,0),(47021,1562745600,39,0,0,0),(62834,156274560 0,40,0,0,0),(48913,1562745600,39,0,0,0),(47187,156 2745600,39,0,0,0),(46054,1562745600,39,0,0,0),(471 36,1562745600,39,0,0,0),(41753,1562745600,39,0,0,0 ),(64957,1562745600,12,0,0,0),(43393,1562745600,40 ,0,0,0),(61904,1562745600,8,45061904,59599203,7560 3496),(58077,1562745600,13,0,0,0),(61016,156274560 0,41,0,0,0),(60459,1562745600,9,0,0,0),(74022,1562 745600,1,193272475648,193272475648,193272475648),( 73997,1562745600,39,4096,18169,46080),(46771,15627 45600,39,0,0,0),(65919,1562745600,8,0,0,0),(36076, 1562745600,39,2318401536,2963060683,3865051136),(6 3791,1562745600,12,1616,6670,23560),(71718,1562745 600,39,0,0,0),(36653,1562745600,39,0,0,0),(40435,1 562745600,39,52098469888,52098510217,52098600960), (47023,1562745600,39,5424,22645,424976),(67212,156 2745600,39,0,0,0),(44910,1562745600,39,0,0,0),(698 35,1562745600,40,0,0,0),(71858,1562745600,39,0,0,0 ),(39259,1562745600,39,2097152,2097152,2097152),(3 6596,1562745600,39,514,514,514),(71120,1562745600, 38,0,0,0),(58786,1562745600,39,0,8,148),(29212,156 2745600,39,0,0,0),(48775,1562745600,78,4,4,4),(612 06,1562745600,42,0,0,0),(60154,1562745600,41,51097 19040,5713319586,5924425728),(72253,1562745600,40, 255,255,255),(66283,1562745600,12,0,0,0),(61209,15 62745600,42,0,0,0),(71039,1562745600,38,0,0,0),(37 482,1562745600,39,4487001230,4487001230,4487001230 ),(69574,1562745600,39,0,0,0),(72684,1562745600,39 ,0,0,0),(38772,1562745600,40,0,31744,1255424),(652 23,1562745600,12,0,0,0),(61373,1562745600,41,0,0,0 ),(50156,1562745600,39,0,0,0),(36260,1562745600,39 ,0,0,0),(67888,1562745600,13,5993880,13501591,3402 4848),(46575,1562745600,39,0,0,0),(36454,156274560 0,39,97,236,395),(69071,1562745600,40,255,255,255) ,(48771,1562745600,78,2,2,2),(72178,1562745600,39, 255,255,255),(54814,1562745600,7,0,0,0),(67130,156 2745600,38,171432,325603,764480),(47564,1562745600 ,39,0,0,0),(37175,1562745600,39,59397544,59398684, 59399824),(47574,1562745600,39,0,0,0),(74195,15627 45600,13,187191296,190831694,197677056),(66152,156 2745600,8,0,0,0),(65626,1562745600,8,0,0,0),(62734 ,1562745600,40,0,0,0),(67729,1562745600,8,0,0,0),( 74871,1562745600,8,0,0,0),(64960,1562745600,12,164 40,77687,138048),(72635,1562745600,39,0,0,0),(6620 2,1562745600,8,0,0,0),(32224,1562745600,13,0,0,0), (76837,1562745600,39,0,0,0),(71591,1562745600,40,0 ,0,0),(61183,1562745600,41,0,0,0),(58068,156274560 0,13,0,0,0),(37808,1562745600,40,0,20,115),(45837, 1562745600,39,0,0,0),(31262,1562745600,40,43920,36 9422,771552),(32266,1562745600,8,0,0,0),(38326,156 2745600,39,258000000,328692307,375000000),(47208,1 562745600,39,0,0,0),(75858,1562745600,39,19,43,61) ,(39841,1562745600,40,0,0,0),(59255,1562745600,13, 844248,905035,968048),(43442,1562745600,40,0,0,0), (57179,1562745600,8,0,0,0),(63033,1562745600,83,2, 2,2),(75628,1562745600,39,0,0,0),(75605,1562745600 ,40,0,0,0),(35165,1562745600,26,0,0,0),(36216,1562 745600,39,25323802624,25324447901,25324851200),(41 557,1562745600,39,0,0,0),(44371,1562745600,39,0,0, 0),(61761,1562745600,13,0,0,0),(73922,1562745600,4 0,1,1,1),(72845,1562745600,40,255,255,255),(61390, 1562745600,42,0,0,0),(60708,1562745600,41,0,0,0),( 47621,1562745600,39,20782649344,20782649344,207826 49344),(69411,1562745600,1,128322629632,1283226296 32,128322629632),(63897,1562745600,8,0,0,0),(68817 ,1562745600,39,0,0,0),(39842,1562745600,40,0,0,0), (63651,1562745600,8,0,0,0),(37053,1562745600,39,79 000000,101000000,184000000),(69531,1562745600,39,0 ,0,0),(40764,1562745600,39,0,0,0),(62391,156274560 0,40,6,6,6),(69864,1562745600,40,0,0,0),(40476,156 2745600,39,150,203,260),(72623,1562745600,39,255,2 55,255),(63692,1562745600,8,0,0,0),(62863,15627456 00,40,0,0,0),(63042,1562745600,83,2,2,2),(51222,15 62745600,13,0,0,0),(45270,1562745600,39,0,0,0),(74 802,1562745600,13,0,0,0),(73913,1562745600,39,0,0, 0),(60838,1562745600,41,0,0,0),(37700,1562745600,3 9,4,4,4),(63611,1562745600,4,0,0,0),(62194,1562745 600,1,4129492992,4129492992,4129492992),(76216,156 2745600,39,0,0,0),(62490,1562745600,40,0,0,0),(688 81,1562745600,8,0,0,0),(43994,1562745600,40,452945 1008,4715379712,4760133632),(41815,1562745600,39,0 ,0,0),(34650,1562745600,39,8589934592,8589934592,8 589934592),(45359,1562745600,39,0,0,0),(58024,1562 745600,13,0,0,0),(77927,1562745600,40,0,856,3744), (76928,1562745600,40,80904,777273,5750528),(77221, 1562745600,40,471859200,823105945,2061500416),(603 25,1562745600,9,0,0,0),(63037,1562745600,83,2,2,2) ,(46414,1562745600,39,0,0,0),(61324,1562745600,42, 4148654080,4148654080,4148654080),(62512,156274560 0,40,0,0,0),(29203,1562745600,39,0,0,0),(76133,156 2745600,39,0,262,10240),(36050,1562745600,39,1,1,1 ),(77161,1562745600,40,0,0,0),(37391,1562745600,40 ,0,281,3072),(31941,1562745600,39,0,0,2),(63853,15 62745600,8,0,0,0),(41843,1562745600,39,0,0,0),(470 22,1562745600,39,5424,22645,424976),(48931,1562745 600,4,0,0,0),(47633,1562745600,39,0,0,0),(44998,15 62745600,39,0,0,0),(57103,1562745600,13,0,0,0),(62 554,1562745600,71,2,2,2),(62367,1562745600,40,0,0, 0),(69297,1562745600,39,0,0,0),(62899,1562745600,4 0,0,0,0),(44361,1562745600,39,0,0,0),(72851,156274 5600,39,255,255,255),(44228,1562745600,40,0,0,0),( 60396,1562745600,9,0,0,0),(64911,1562745600,8,0,0, 0),(77700,1562745600,40,0,0,0),(74081,1562745600,7 7,0,0,0),(44775,1562745600,39,0,0,0),(69931,156274 5600,39,255,255,255),(63934,1562745600,12,8712,951 8,11520),(59717,1562745600,9,0,0,0),(46397,1562745 600,39,0,0,0),(59774,1562745600,9,0,0,0),(37711,15 62745600,39,4143972352,4147010533,4150263808),(726 98,1562745600,39,0,0,0),(76588,1562745600,39,180,2 90,702),(44315,1562745600,39,0,0,0),(76954,1562745 600,39,0,0,0),(61198,1562745600,41,0,0,0),(66250,1 562745600,12,0,0,0),(75150,1562745600,39,255,255,2 55),(61730,1562745600,8,0,0,0),(77423,1562745600,3 9,1371537408,1373876539,1376780288),(40302,1562745 600,39,0,0,1),(46517,1562745600,39,21304,24174,295 84),(36494,1562745600,40,118,306,2831),(77014,1562 745600,39,0,0,0),(60452,1562745600,9,0,0,0),(63291 ,1562745600,236,1,1,1)(65481,1562745600,8,0,0,0),( 58820,1562745600,39,0,0,0),(77898,1562745600,1,429 0768896,4290768896,4290768896),(60301,1562745600,9 ,0,0,0),(41554,1562745600,39,0,0,0),(69335,1562745 600,39,0,0,0),(37421,1562745600,39,1024,4201,9216) ,(39091,1562745600,39,1287651328,1515837597,180355 0720),(59106,1562745600,13,548600,795810,1546568), (39227,1562745600,39,170917888,373750127,729808896 ),(76864,1562745600,40,0,0,0),(57823,1562745600,8, 0,0,0),(47082,1562745600,39,0,0,0),(61678,15627456 00,8,0,0,0),(73491,1562745600,39,0,0,0),(57200,156 2745600,8,0,0,0),(74939,1562745600,13,0,0,0),(5482 3,1562745600,7,0,0,0),(40353,1562745600,39,1050671 34976,105067193790,105067266048),(69296,1562745600 ,39,4472,44256,338776),(61922,1562745600,13,0,0,0) ,(59642,1562745600,14,0,0,0),(45432,1562745600,40, 2207145984,2681006182,2773286912),(44057,156274560 0,39,0,0,0),(76038,1562745600,39,2167406592,216764 8571,2168455168),(32125,1562745600,8,0,0,0),(69631 ,1562745600,79,0,0,0),(69980,1562745600,39,6164473 4464,61717973464,61765320704),(70332,1562745600,79 ,2,2,2),(71881,1562745600,39,1,1,1),(71518,1562745 600,39,0,0,0),(59752,1562745600,9,0,0,0),(29730,15 62745600,39,3166,3298,3441),(36392,1562745600,39,0 ,0,0),(59143,1562745600,8,0,0,0),(76667,1562745600 ,39,46080,121645,472064),(61159,1562745600,42,0,0, 0),(51176,1562745600,8,0,0,0),(60680,1562745600,42 ,0,0,0),(62375,1562745600,40,0,0,0),(70053,1562745 600,78,2,2,2),(32440,1562745600,13,15160,18022,206 16),(66008,1562745600,8,0,0,0),(38367,1562745600,3 9,0,0,0),(60701,1562745600,41,0,0,0),(61442,156274 5600,42,0,0,0),(65828,1562745600,12,0,894,4520),(3 7718,1562745600,39,1,1,1),(47105,1562745600,40,0,0 ,0),(60898,1562745600,41,0,0,0),(56902,1562745600, 8,0,0,0),(37804,1562745600,40,408,1447,5972),(5734 8,1562745600,13,9696,2367196,24578392),(67830,1562 745600,8,0,0,0),(77541,1562745600,39,5348290560,53 48290560,5348290560),(59775,1562745600,9,0,0,0),(3 5159,1562745600,6,2,2,2),(77492,1562745600,39,0,28 8,4096),(72409,1562745600,39,255,255,255),(69904,1 562745600,40,0,0,0),(44661,1562745600,39,0,0,0),(7 7008,1562745600,39,0,0,0); " 29514:20190710:103929.551 syncing trend data done 29514:20190710:103929.624 Zabbix Server stopped. Zabbix 4.2.1 (revision 92832). What very weird is, if i stop and start the zabbix server, i don't see new text in de error.log.. i have also a attachement of systemctl status zabbix-server what can i do? Many thanks for helping! |
| Comments |
| Comment by Kaspars Mednis [ 2019 Jul 10 ] |
|
Hello Robert, You value cache seems to low, so you must tune your cache settings. Please take a look into Zabbix server conf file, and increase size of your value cache and other caches if needed. There is a nice graph on Zabbix server host called Zabbix cache usage, where you can see how much your caches are used. If they are less than 75% free, you must increase the corresponding cache size in zabbix_server.conf file and restart your Zabbix server. At the moment i can not see any bug here, seems more like a performance tuning issue. I will wait for a response from your side and close this issue. Please note, this is a bug tracker not a free support portal, if you have any questions about usage of Zabbix please use our forum for a free help from our community members or you can use our professional services Best regards, |
| Comment by Robbert van der Kooi [ 2019 Jul 10 ] |
|
Hello Kaspers, Many thanks for reply, i have tuned the setting to this: ### Option: CacheSize # Size of configuration cache, in bytes. # Shared memory size for storing host, item and trigger data. # # Mandatory: no # Range: 128K-8G # Default: CacheSize=2G and: ### Option: ValueCacheSize # Size of history value cache, in bytes. # Shared memory size for caching item history data requests. # Setting to 0 disables value cache. # # Mandatory: no # Range: 0,128K-64G # Default: ValueCacheSize=2G I have restart about 6 hours ago the server, buth without any positive result. Do you have a another idea? Many thanks. |
| Comment by Kaspars Mednis [ 2019 Jul 11 ] |
|
2G cache size may be an overkill, i do not know your previous values, but what do you mean by without any positive result ? Please attach the Zabbix Server log file.
|
| Comment by Robbert van der Kooi [ 2019 Jul 11 ] |
|
Yes, i think i do it good.. original size was default.. i mean with witouh any positive result that the server won't start with any log. How can i send the log file? the file is 240MB. I have copied the latst rows from the file and attachement here. |
| Comment by Kaspars Mednis [ 2019 Jul 11 ] |
|
if the server is not starting at all, you can issue the zabbix_server command from the commandline, it will show the reason.
|
| Comment by Robbert van der Kooi [ 2019 Jul 11 ] |
|
Which command you mean?
root@zabbix:~# systemctl status zabbix-server Jul 10 23:24:25 zabbix systemd[1]: Starting LSB: Start zabbix-server daemon... |
| Comment by Kaspars Mednis [ 2019 Jul 11 ] |
|
just zabbix_server command, it will try to launch the binary and will show you any problems with current config if you have such. |
| Comment by Robbert van der Kooi [ 2019 Jul 11 ] |
|
sorry i feel my very stupid now.. i get this:
root@zabbix:~# zabbix_server |
| Comment by DaneT [ 2019 Jul 11 ] |
|
Please try to use /usr/sbin/zabbix_server command then. Also - could you please check that zabbix server configuration file |
| Comment by Robbert van der Kooi [ 2019 Jul 11 ] |
|
Hello, Many thanks for helping! the files are completed removed from /usr/bin/zabbix_server I have reinstalled zabbix-server and everyting is working now. Thanks. |