[ZBX-15677] module.h is insufficient to compile loadable module Created: 2019 Feb 19  Updated: 2024 Apr 10

Status: Reopened
Project: ZABBIX BUGS AND ISSUES
Component/s: Agent (G), Proxy (P), Server (S)
Affects Version/s: 2.2.24rc1, 3.0.26rc1, 4.0.6rc1, 4.2.0beta1
Fix Version/s: None

Type: Problem report Priority: Trivial
Reporter: Glebs Ivanovskis Assignee: Zabbix Support Team
Resolution: Unresolved Votes: 4
Labels: compilation, loadablemodule, patch
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Attachments: File dummiest.c    
Team: Team C
Sprint: Sprint 56 (Sep 2019), Sprint 55 (Aug 2019), Sprint 51 (Apr 2019), Sprint 54 (Jul 2019), Sprint 57 (Oct 2019), Sprint 58 (Nov 2019), Sprint 59 (Dec 2019), Sprint 60 (Jan 2020), Sprint 61 (Feb 2020), Sprint 62 (Mar 2020), Sprint 63 (Apr 2020), Sprint 64 (May 2020), Sprint 65 (Jun 2020), Sprint 66 (Jul 2020), Sprint 67 (Aug 2020)
Story Points: 0.25

 Description   

Steps to reproduce:

  1. Download Zabbix sources or check out from repository
  2. Navigate to src/modules/dummy
  3. Comment out
    #include "sysinc.h"

    leaving just

    #include "module.h"
  4. Try to build module: $ make

Result:
Tons of compilation errors.

Expected:
Successful compilation.

Speculations:
This happens because module.h depends on zbxtypes.h, and zbxtypes.h in turn (implicitly!) depends on lots of system headers. So the only way to compile loadable module is to

#include "sysinc.h"

before(!)

#include "module.h"

But that's not the end of the story because sysinc.h includes system headers conditionally with conditional variables defined in another header(!!!) config.h which is generated by configure script. So one has to run $ ./configure before even attempting to build poor dummy module.

I think this is unacceptable. Module authors need to be quite creative to work around that and this puts extra stress on build systems and continuous integration. The only reason to

#include "zbxtypes.h"

in module.h is definition of zbx_uint64_t:

#if defined(_WINDOWS)
/* ... (irrelevant while loadable modules are only available on UNIX platforms) */
#else	/* _WINDOWS */
/* ... */
#	define zbx_uint64_t	uint64_t
/* ... */
#endif	/* _WINDOWS */

My suggestion is to provide definition of zbx_uint64_t in module.h if it is used outside of Zabbix:

Index: include/module.h
===================================================================
--- include/module.h	(revision 89838)
+++ include/module.h	(working copy)
@@ -20,7 +20,10 @@
 #ifndef ZABBIX_MODULE_H
 #define ZABBIX_MODULE_H
 
-#include "zbxtypes.h"
+#ifndef zbx_uint64_t
+#	include <stdint.h>
+#	define zbx_uint64_t uint64_t
+#endif
 
 #define ZBX_MODULE_OK	0
 #define ZBX_MODULE_FAIL	-1

Since stdint.h is part of C99, this should cover most of use cases.

With explicit inclusion of necessary headers in dummy.c:

Index: src/modules/dummy/dummy.c
===================================================================
--- src/modules/dummy/dummy.c	(revision 89838)
+++ src/modules/dummy/dummy.c	(working copy)
@@ -17,7 +17,10 @@
 ** Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
 **/
 
-#include "sysinc.h"
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+
 #include "module.h"
 
 /* the variable keeps timeout setting for item processing */

the module compiles cleanly requiring just module.h from Zabbix SVN or source tarball. Zabbix code compiles too.



 Comments   
Comment by Michael Veksler [ 2019 Apr 24 ]

The proposed patch has been tested for the following systems:

title result
Ubuntu16.04
Centos6x32
Cento7x64
OpenBSD
FreeBSD
NetBSD
Solaris 10
Solaris 11

to apply the solution: 0.25 sp

Conclusion:

  • : this can be useful when regular 'admin' download some module from GitHub and want to compile and install it on server. This patch will make compilation process more easy and we will become more friendly to our users. We can skip the step ./configure -q
  • : if in the future we decide to provide more functionality, for example "memory usage control of external modules" and show it as internal metrics. As a result, we apparently will depend from zbxtypes.h anyway
Comment by Vladislavs Sokurenko [ 2019 Apr 26 ]

(1) [S] Thank you for the patch cyclone! I suggest style fix, module.h when not used in modules will never have #ifndef zbx_uint64_t true, it will only be true when compiled with dummy.c so moved to dummy.c also zbxtypes.h already have uint64_t defined in zbxtypes.h so left as it is.

diff --git a/src/modules/dummy/dummy.c b/src/modules/dummy/dummy.c
index 532a1a60cf..01af25d81d 100644
--- a/src/modules/dummy/dummy.c
+++ b/src/modules/dummy/dummy.c
@@ -17,7 +17,11 @@
 ** Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA  02110-1301, USA.
 **/
 
-#include "sysinc.h"
+#include <stdlib.h>
+#include <string.h>
+#include <time.h>
+#include <stdint.h>
+
 #include "module.h"
 
 /* the variable keeps timeout setting for item processing */

vso RESOLVED in 2ec803ed96c
MVekslers I agree - it looks preferable
CLOSED

Comment by Michael Veksler [ 2019 Apr 30 ]

Available in:

  • 4.0.8rc1 f7037820a3c
  • 4.2.2rc1 cbbc3135cf8
  • 4.4.0alpha1(trunk) f7037820a3c
Comment by Glebs Ivanovskis [ 2019 May 02 ]

The issue has not been addressed. Once again, the issue is the following: "module.h is insufficient to compile loadable module". Any loadable module. I used dummy.c just as an example of loadable module source. I expect module.h to be fixed, but I see that only dummy.c has been changed.

Comment by Michael Veksler [ 2019 May 02 ]

The main point was in possibility to build any modules without "configure" src tarball. What is the reason for removing dependency on "zbxtypes.h" ?
Because "zbxtypes.h" does not imply an additional dependency and contains all the necessary.

Comment by Glebs Ivanovskis [ 2019 May 02 ]

Please read Description of the issue to the end.

Comment by Glebs Ivanovskis [ 2019 May 02 ]

What is the reason for removing dependency on "zbxtypes.h" ?

Why would I need to download two header files instead of one to compile a module? It is way easier to compile a module when module.h is self-sufficient.

Because "zbxtypes.h" does not imply an additional dependency and contains all the necessary.

I'm not so sure about it...

typedef off_t	zbx_offset_t;

Where is off_t defined? It implicitly requires sys/types.h.

Comment by Glebs Ivanovskis [ 2019 May 31 ]

The goal of this ticket was to make modules like the attached dummiest.c to compile. Here is where we currently stand:

zabbix/src/modules/dummiest $ make
gcc -fPIC -shared -o dummiest.so dummiest.c -I../../../include
In file included from ../../../include/module.h:23:0,
                 from dummiest.c:1:
../../../include/zbxtypes.h:144:9: error: unknown type name ‘uint32_t’
 typedef uint32_t zbx_uint32_t;
         ^~~~~~~~
../../../include/zbxtypes.h:150:9: error: unknown type name ‘off_t’
 typedef off_t zbx_offset_t;
         ^~~~~
../../../include/zbxtypes.h:98:23: error: unknown type name ‘uint64_t’
 # define zbx_uint64_t uint64_t
                       ^
../../../include/zbxtypes.h:181:2: note: in expansion of macro ‘zbx_uint64_t’
  zbx_uint64_t lo;
  ^~~~~~~~~~~~
../../../include/zbxtypes.h:98:23: error: unknown type name ‘uint64_t’
 # define zbx_uint64_t uint64_t
                       ^
../../../include/zbxtypes.h:182:2: note: in expansion of macro ‘zbx_uint64_t’
  zbx_uint64_t hi;
  ^~~~~~~~~~~~
../../../include/zbxtypes.h:98:23: error: unknown type name ‘uint64_t’
 # define zbx_uint64_t uint64_t
                       ^
../../../include/module.h:54:2: note: in expansion of macro ‘zbx_uint64_t’
  zbx_uint64_t lastlogsize;
  ^~~~~~~~~~~~
../../../include/zbxtypes.h:98:23: error: unknown type name ‘uint64_t’
 # define zbx_uint64_t uint64_t
                       ^
../../../include/module.h:81:2: note: in expansion of macro ‘zbx_uint64_t’
  zbx_uint64_t lastlogsize; /* meta information */
  ^~~~~~~~~~~~
../../../include/zbxtypes.h:98:23: error: unknown type name ‘uint64_t’
 # define zbx_uint64_t uint64_t
                       ^
../../../include/module.h:82:2: note: in expansion of macro ‘zbx_uint64_t’
  zbx_uint64_t ui64;
  ^~~~~~~~~~~~
../../../include/zbxtypes.h:98:23: error: unknown type name ‘uint64_t’
 # define zbx_uint64_t uint64_t
                       ^
../../../include/module.h:149:2: note: in expansion of macro ‘zbx_uint64_t’
  zbx_uint64_t itemid;
  ^~~~~~~~~~~~
../../../include/zbxtypes.h:98:23: error: unknown type name ‘uint64_t’
 # define zbx_uint64_t uint64_t
                       ^
../../../include/module.h:158:2: note: in expansion of macro ‘zbx_uint64_t’
  zbx_uint64_t itemid;
  ^~~~~~~~~~~~
../../../include/zbxtypes.h:98:23: error: unknown type name ‘uint64_t’
 # define zbx_uint64_t uint64_t
                       ^
../../../include/module.h:161:2: note: in expansion of macro ‘zbx_uint64_t’
  zbx_uint64_t value;
  ^~~~~~~~~~~~
../../../include/zbxtypes.h:98:23: error: unknown type name ‘uint64_t’
 # define zbx_uint64_t uint64_t
                       ^
../../../include/module.h:167:2: note: in expansion of macro ‘zbx_uint64_t’
  zbx_uint64_t itemid;
  ^~~~~~~~~~~~
../../../include/zbxtypes.h:98:23: error: unknown type name ‘uint64_t’
 # define zbx_uint64_t uint64_t
                       ^
../../../include/module.h:176:2: note: in expansion of macro ‘zbx_uint64_t’
  zbx_uint64_t itemid;
  ^~~~~~~~~~~~
../../../include/zbxtypes.h:98:23: error: unknown type name ‘uint64_t’
 # define zbx_uint64_t uint64_t
                       ^
../../../include/module.h:185:2: note: in expansion of macro ‘zbx_uint64_t’
  zbx_uint64_t itemid;
  ^~~~~~~~~~~~
Makefile:2: recipe for target 'dummiest' failed
make: *** [dummiest] Error 1

If you add

#include "sysinc.h"

on the top as suggested by documentation, the module can be successfully compiled and loaded by Zabbix daemon. But given how many extra steps are needed to produce sysinc.h this solution is far from perfect.

Making module.h self-containing will enable Zabbix to provide development packages, which will be a huge leap forward in terms of ease of module development and deployment.

Comment by Michael Veksler [ 2019 Sep 06 ]

What do you propose to do with already written plugins. They can use different values from zbxtypes.h. Example https://github.com/LMacPhail/zabbix-history-influxdb
I would not want to break backward compatibility without any reason, such as introducing a new API - for example.

Comment by Glebs Ivanovskis [ 2019 Sep 06 ]

What do you propose to do with already written plugins.

Nothing. I have proposed a non-breaking change in the Description. Module you've given as an example compiles fine with patched Zabbix.

They can use different values from zbxtypes.h.

Technically, they can. However, they probably shouldn't. Zabbix isn't explicit about this, unfortunately, but modules should not use anything beyond what's described in documentation. See ZBX-11813 for that matter.

I would not want to break backward compatibility without any reason...

Is dimir spreading this disease? There is a reason... I think, a good one — improving Zabbix.

...introducing a new API - for example.

Could you please elaborate. Why would you need a new API to address the issue?

Comment by Michael Veksler [ 2019 Dec 03 ]

I tested the current version, where only 2 files are required (instead of 1 from your suggestion)
All works

ls -l ./dummy/*
-rw-rw-r-- 1 mvekslers mvekslers 12871 jūn  3  2019 ./dummy/dummy.c
-rwxrwxr-x 1 mvekslers mvekslers 13272 dec  3 16:45 ./dummy/dummy.so
-rw-rw-r-- 1 mvekslers mvekslers    73 jūn  3  2019 ./dummy/Makefile
-rw-rw-r-- 1 mvekslers mvekslers  5233 jūl  9 13:38 ./dummy/module.h
-rw-rw-r-- 1 mvekslers mvekslers   245 jūn  3  2019 ./dummy/README
-rw-rw-r-- 1 mvekslers mvekslers  5135 dec  3 16:29 ./dummy/zbxtypes.h

Maybe this is enough?

Comment by FIll [ 2019 Dec 03 ]

good

Comment by Glebs Ivanovskis [ 2019 Dec 03 ]

MVekslers, try to compile dummiest.c. Currently it only includes module.h, let's see what changes you'l have to make and then discuss if they can be justified.

Comment by Michael Veksler [ 2019 Dec 04 ]

Can I add system libs (not internal !):

#include <stdlib.h>
#include <stdint.h>

?

Comment by Glebs Ivanovskis [ 2019 Dec 04 ]

In my opinion it will look silly. In addition it may cause issues with tools like clang-format, which try to order headers alphabetically.

Have you seen a library which asks developer to include a couple of standard headers before including library's header? Why can't module.h include these standard headers?

Comment by Glebs Ivanovskis [ 2020 Mar 20 ]

Hello, MVekslers! I hope you are doing well.

Where are we stuck with this issue? Do you have any updates?

Comment by Michael Veksler [ 2020 Mar 20 ]

Hi cyclone, thanks for the question. As I understand it we have solved the main problem and to compile a plugin is not necessary to call 'configure' more. Other changes to module.h are not sufficiently motivated, especially if we take into account that module.h is also involved in GO part.

Comment by Glebs Ivanovskis [ 2020 Mar 20 ]

MVekslers, thank you for a prompt response!

As I understand it we have solved the main problem and to compile a plugin is not necessary to call 'configure' more.

Which "plugin" do you have in mind? dummy.c provided with Zabbix source or any plugin (I'd prefer to say "loadable module") including dummiest.c? If by "solved" you mean adding two headers for no reason, then it should be documented.

Comment by Michael Veksler [ 2020 Mar 20 ]

I agree with you, this should be documented. Thanks for the efforts  and your ideas.

Comment by Glebs Ivanovskis [ 2020 Mar 20 ]

Good. I hope it won't take long.

Comment by Glebs Ivanovskis [ 2020 Aug 18 ]

How naive I was...

Comment by Michael Veksler [ 2020 Aug 18 ]

You are not entirely right . Correct sample was placed in the documentation long time ago. I thought the description for module.h was corrected too, but you right - it was skipped. I will do the correction of (2.5) module.h description.

Thanks for help.

Comment by Michael Veksler [ 2020 Aug 18 ]

The correction of (2.5) for module.h  was done !

Comment by Glebs Ivanovskis [ 2020 Aug 18 ]

Finally! Should I close the ticket or will you take care of it?

Comment by Michael Veksler [ 2020 Aug 18 ]

Only You !

Comment by Glebs Ivanovskis [ 2021 Aug 07 ]

At some point the issue was resolved:

As I understand it we have solved the main problem and to compile a plugin is not necessary to call 'configure' more.

But it turns out that it didn't last long. ZBX-16998 added #include "sysinc.h" into zbxtypes.h and once again it is necessary to do ./configure in order to build any kind of loadable module. Otherwise compiler can't find config.h (included from sysinc.h, included from zbxtypes.h, included from module.h).

The most frustrating part is that I closed the ticket when it had already been broken. I did not double-check, my bad. Reopening now.

Please reconsider the patch that was originally suggested (see Description). However you decide to fix it, this time please add a test to stop the issue from reappearing again.

Comment by Glebs Ivanovskis [ 2022 Feb 23 ]

Dear mvekslers, vso, is there any update?

Comment by Glebs Ivanovskis [ 2023 Mar 13 ]

Dear mvekslers, vso, sorry for bothering you once in a year. Is there any update?

Generated at Sat Apr 27 03:47:54 EEST 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.