Recently the regular expression processing was extended to include better error handling. However it was done by two separate issues touching two paths of regular expression processing (ZBXNEXT-1638, ZBXNEXT-1250) and as result has a different approach:

static char	*zbx_regexp(const char *string, const char *pattern, int *len, int flags)
and
static int	regexp_sub(const char *string, const char *pattern, const char *output_template, int flags, char **out)

zbx_regexp() returns pointer to matched string or NULL if the string doesn't match or the pattern was invalid. If NULL is returned, then len contains either SUCCEED (pattern was valid) or FAIL (otherwise).

regexp_sub() returns FAIL if pattern is invalid and SUCCEED otherwise. If SUCSEED is returned then out contains either substituted output string or NULL if there is no match.

In reality the zbx_regexp() len parameter is used only to return error code, so we can simply use similar signature to regexp_sub() function:

static int	zbx_regexp(const char *string, const char *pattern, int flags, char **output)

Also complex return codes usually aren't used in Zabbix. It might be better to return simply SUCCEED/FAIL based on regular expression compilation success (with optional error message output parameter) and use additional output parameter to return match/not match.

Functions zbx_regexp() and regexp_sub() (especially with changes proposed by wiper) will be virtually identical. I think it is a good idea to merge them into one.

However, there are difficulties with higher level functions. Suggested solution (SUCCEED/FAIL + NULL/matched string) won't be particularly convenient in regexp_match_ex() where we are not interested in matched string but only in match/not match.

Maybe this feature request can be reviewed now - after regexp support has been moved to libpcre.
regexp_compile() reports in error message why the regexp does not compile but throws away position number where the error was detected.
regexp_compile() can be improved to report position, e.g.

\ at end of pattern at position 11

in case

/home/zabbix40/ZBX-15090/test.log.*\

was given as the first parameter to logrt[].
However this requires adding deallocation of error messages in callers (currently regexp_compile() returns pointer to error message in static memory).