-
Problem report
-
Resolution: Duplicate
-
Critical
-
None
-
4.0.2
-
None
-
RHEL 7 VM, MariaDB 5.5.52
Last night, I upgraded my company's Zabbix installations from 3.4.13 to 4.0.2. This morning, one of our Zabbix installations has been alarming that it's out of disk space.
The cause of this is that zabbix_server.log is rapidly filling up the disk. Every 10 seconds, it is attempting to run a ridiculously long query, and when the query fails, it logs the whole query while claiming that MySQL is down (MySQL is not down; the query is failing because it's unreasonably long). Each time it logs the query, the log entry containing the query is nearly four megabytes long.
Since then, I've set the maximum log file size to 128MB so it doesn't completely fill up /var while I'm trying to debug this, but the situation is still untenable, and I need to know how to stop this query from running.
The query is a long select from problem_tag... here are the first 4096 characters from one log entry (I cannot paste the whole thing here:
59356:20181205:154144.373 [Z3005] query failed: [2006] MySQL server has gone away [select eventid,tag,value from problem_tag where (eventid between 80530 and 80535 or eventid between 739271 and 739277 or eventid between 986879 and 986886 or eventid between 986888 and 986898 or eventid between 986900 and 986906 or eventid between 986920 and 986926 or eventid between 986948 and 986952 or eventid between 986977 and 986983 or eventid between 986986 and 986996 or eventid between 987011 and 987017 or eventid between 987032 and 987037 or eventid between 987047 and 987066 or eventid between 987071 and 987076 or eventid between 987087 and 987094 or eventid between 987097 and 987106 or eventid between 987111 and 987118 or eventid between 987132 and 987138 or eventid between 987158 and 987162 or eventid between 987187 and 987192 or eventid between 987195 and 987204 or eventid between 987225 and 987229 or eventid between 987245 and 987250 or eventid between 987256 and 987276 or eventid between 987296 and 987321 or eventid between 987336 and 987341 or eventid between 987361 and 987365 or eventid between 987396 and 987401 or eventid between 987404 and 987415 or eventid between 987428 and 987434 or eventid between 987449 and 987454 or eventid between 987456 and 987464 or eventid between 987466 and 987479 or eventid between 987484 and 987488 or eventid between 987499 and 987523 or eventid between 987544 and 987549 or eventid between 987567 and 987571 or eventid between 987593 and 987599 or eventid between 987602 and 987612 or eventid between 987626 and 987632 or eventid between 987648 and 987653 or eventid between 987655 and 987662 or eventid between 987664 and 987678 or eventid between 987683 and 987688 or eventid between 987697 and 987714 or eventid between 987718 and 987724 or eventid between 987738 and 987744 or eventid between 987807 and 987812 or eventid between 987831 and 987837 or eventid between 987852 and 987857 or eventid between 987862 and 987871 or eventid between 987873 and 987883 or eventid between 987888 and 987892 or eventid between 987902 and 987927 or eventid between 987940 and 987946 or eventid between 987966 and 987970 or eventid between 987996 and 988002 or eventid between 988005 and 988015 or eventid between 988028 and 988034 or eventid between 988056 and 988068 or eventid between 988070 and 988079 or eventid between 988084 and 988089 or eventid between 988101 and 988110 or eventid between 988112 and 988124 or eventid between 988138 and 988144 or eventid between 988164 and 988168 or eventid between 988194 and 988199 or eventid between 988202 and 988213 or eventid between 988226 and 988232 or eventid between 988246 and 988251 or eventid between 988253 and 988275 or eventid between 988280 and 988285 or eventid between 988294 and 988310 or eventid between 988312 and 988319 or eventid between 988334 and 988340 or eventid between 988360 and 988364 or eventid between 988389 and 988394 or eventid between 988397 and 988407 or eventid between 988412 and 988420 or eventid between 988432 and 988437 or eventid between 988450 and 988468 or eventid between 988473 and 988477 or eventid between 988489 and 988513 or eventid between 988535 and 988540 or eventid between 988554 and 988560 or eventid between 988579 and 988587 or eventid between 988592 and 988602 or eventid between 988616 and 988622 or eventid between 988638 and 988643 or eventid between 988645 and 988667 or eventid between 988672 and 988677 or eventid between 988686 and 988691 or eventid between 988694 and 988706 or eventid between 988708 and 988713 or eventid between 988728 and 988734 or eventid between 988754 and 988758 or eventid between 988793 and 988803 or eventid between 988816 and 988822 or eventid between 988836 and 988841 or eventid between 988843 and 988865 or eventid between 988870 and 988875 or eventid between 988884 and 988908 or eventid between 988923 and 988929 or eventid between 988949 and 988953 or eventid between 988982 and 988986 or eventid between 988989 and 988999 or eventid between 989015 and 989019 or eventid between 989047 and 989052 or
This is approximately one thousandth the length of the full query.
And then it will typically log something like:
59356:20181205:155332.918 database is down: retrying in 10 seconds 59356:20181205:155342.919 database connection re-established
(again, the database isn't down; Zabbix just thinks it is because it's failing to run this unreasonable query)
Note that we have two Zabbix installations; only one of them is acting like this.
If you need more details, I'm attaching one of the logs which I grabbed about a minute or so after restarting Zabbix.
Thank you.
- duplicates
-
-
- Closed
-