The problematic function is evaluate_simple() in src/libs/zbxserver/expression.c.
The way it works is as follows. When it processes an operator, it searches the expression for that operator from left to right, splits the expression in two, recursively evaluates each part, and, finally, applies the operator.
This algorithm has problems for both "-" and "/". For instance, evaluating "64/8/8" yields 64, whereas the correct answer is 1. Similarly, "100-10-1" would yield "91", whereas the correct value is 89. Luckily, evaluate_simple() evaluates "100-10-1" to 89, because of the function compress_signs(), which turns "100-10-1" into "100+N10+N1".
A quick solution to this problem would be to search for "-" and "/" from right to left (using strrchr), instead of left to right (with strchr).
Another solution would be to completely rewrite evaluate() and evaluate_simple(), as more efficient algorithms for evaluating expressions exist.
PS: You can also note the statement *p='|'; throughout the function, executed for all the operators. It does not seem to have any adverse effects, however, it seems to be a sign of copy-paste, which, potentially, is a major source of problems of its own.