as an example, https://www.zabbix.com/documentation/2.2/manual/installation/requirements uses 50/50 "back-end" and "back end".

personally i'd even prefer "backend", which is fairly common in IT.

whichever it is, it should be unified across the various branches in the manual, and the decision should be documented in https://www.zabbix.org/wiki/Docs/specs/syntax