[ZBX-9347] Information about quoting incoming file for zabbix_sender utility Created: 2015 Feb 27  Updated: 2017 May 30  Resolved: 2017 Feb 14

Status: Closed
Project: ZABBIX BUGS AND ISSUES
Component/s: Documentation (D)
Affects Version/s: 2.2.8, 2.4.4
Fix Version/s: 2.2.11rc1, 2.4.7rc1, 3.0.0alpha4

Type: Incident report Priority: Blocker
Reporter: Alexey Pustovalov Assignee: Unassigned
Resolution: Fixed Votes: 0
Labels: sender
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
Team: Team A
Sprint: Sprint 1

 Description   

If objects have whitespaces, these objects must be quoted using double quotes.



 Comments   
Comment by Alexander Vladishev [ 2015 Mar 04 ]

Related issue: ZBX-1347

Comment by Alexander Vladishev [ 2015 Mar 04 ]

zabbix_sender input file format:

      ,------------,
-->+->|   value    |->+-->
   ^  '------------'  |
   |  ,------------,  |
   '--| whitespace |<-'
      '------------'


value
      ,---------------,
-->+->| quoted string |->+-->
   |  '---------------'  ^
   |  ,---------------,  |
   '->|    string     |--'
      '---------------'


whitespace

   ,---------------------,
   V     ,---------,     |
-->+->+->| <SPACE> |->+->+-->
      |  '---------'  ^
      |  ,---------,  |
      '->|  <TAB>  |--'
         '---------'


quoted string
          ,------------------------------------,
          |  ,------------------------------,  |
          |  |     ,------------------,     |  |
   ,---,  |  V     |   Any UNICODE    |     |  V  ,---,
-->| " |->+->+->+->| character except |->+->+->+->| " |-->
   '---'        |  |  (") and (<LF>)  |  ^        '---'
                |  '------------------'  |
                |      ,---,  ,---,      |
                +----->| \ |--| " |------+
                |      '---'  '---'      |
                |      ,---,  ,---,      |
                '----->| \ |--| n |------'
                       '---'  '---'

string
   ,-----------------------------,
   |  ,-----------------------,  |
   V  |      Any UNICODE      |  |
-->+->|   character except    |->+-->
      | (<SPACE>) and (<TAB>) |
      '-----------------------'
Comment by Martins Valkovskis [ 2015 Mar 31 ]

(1) Documented in:

<richlv> the current note seems to be a generic one, it does not seem to be clear that doublequoting is supposed to be used in the input file - maybe we can clarify that a bit

martins-v It's true, but we are not mentioning an "input file" on that page before. So I wonder if it is ok to start talking about it there.

<richlv> well, it's mentioned in a couple of notes like "If sending many values from an input file..." and "All entries from an input file...". but yes, we don't even mention other things about the input file or even which flag to use for it. maybe it's better to leave all that for the manpage then, and just link to it (like we do already) ?

<dimir> Here's what I did for 2.2 . When/If you are happy with that I could add that info to 2.4 and 3.0 also.

  • added few more details to sender page
  • separated with/without timestamps description on sender page
  • added examples (with/without timestamps) of an input file to sender page
  • added quoting examples to sender page
  • moved note about timestamp adjustment from trapper to sender page
  • removed all information about zabbix_sender from trapper page, just left the link there

Changed pages:

What else could be done:

  • Move sentence "Zabbix sender accepts strings in UTF-8 encoding (for both UNIX-like systems and Windows)." from the manual to the man page (rules)?
  • Move note about timestamp adjustment from the manual to the man page?

martins-v I would move the first sentence to man page, and leave the second in the manual because it's like a manual-style background information.

Overall, nice to see that much more information is made available to users. Great work!

<dimir> Moved UTF-8 to the man page. I agree on keeping timestamp adjustment in the manual. One more thing, currently we have a note about sending up to 250 entries in one go in both the man page and online manual. What do you see as best place to keep it?

<Rich> manpage

<dimir> So, in addition:

  • moved note about UTF-8 from online manual to the man page
  • same for sequence of sending values top-down
  • info about batching values up to 250 left only in the man page

RESOLVED in r55882 (note, due to the branch recreation you'd have to remove the old checkout and pull the new one)

<richlv>

  • there seems to be a singular/plural mixup: "The contents of the file is accepted...". also, maybe it all could be simplified to:
    "Contents of the input file must be in the UTF-8 encoding" ? no need to explicitly list operating systems here, we don't do that for other features.
  • note on data being sent sequentially is now duplicated in the paragraph above the example line and in the notes below
  • maybe it's better to move these unconnected notes in the same paragraph (above the example line) ? we could also get rid of "Note, that"

<dimir> Thanks, great suggestions. Regarding the last one, I really like the example to be right after simple description and after that list all the special things. Please see how I did it, also added a couple of other changes.

RESOLVED in r56225

<richlv> ok, looks real good now, just last minor nitpick
"Contents of the input file must be .... All values from an input file are sent..." - let's use 'the' in the second case as well

<dimir> I'm sure it'd be quicker if you'd do all the changes yourself and I just review, your language skills are much better than mine. Just in case there are more things to improve.

RESOLVED in r56230

<richlv> we were so close already, and by now it all looks good to me - thank you
CLOSED

Comment by richlv [ 2015 Mar 31 ]

(2) should this be mentioned in the manpage ? it currently talks about doublequoting empty values in the example section

<dimir> After discussion with martins-v and richlv we decided it's best to describe in detail the input file format in the man page, generate html version from it and then just link to it from anywhere.

Development branch for 2.0: svn://svn.zabbix.com/branches/dev/ZBX-9347

RESOLVED in r53223

a) in "to use hostname from configuration file" let's be specific that it's "agent hostname"
b) in "If an entry contains special characters", let's say that it's doublequotes instead of spec chars (but see e)
c) it says "A backslash itself must not be escaped", but it seems that it must be - but only inside double quotes, otherwise - not.
d) "In case the last character is a backslash double-quotes cannot be used" - so how to send a value (ignoring single quotes) 'a \' ?
e) in general, quoting requirements seem to be weird, so we might have to discuss them more

<dimir> Based on more discussion on the subject I've added the following changes:

  • rules
  • use cases
  • notes

Please review. RESOLVED in r53267

<richlv> discussed in more detail in other subissues, this subissue
CLOSED

Comment by dimir [ 2015 Apr 20 ]

richlv, would it be possible for you to review the (2)?

The changes are:

       -i, --input-file <inputfile>

              Load values from input file. Specify - to read values from standard input.

              Each line of the input file must contain 3 whitespace delimited entries: <hostname> <key> <value>. Specify - as <hostname> to use hostname from configuration file (specified with --config) or --host argument. All
              entries are sent in a sequential order top-down.

              An example of a line of the input file:

               host1 item1 sometext

              If an entry contains whitespace it must be double-quoted (escaping a whitespace will not work):

               "Zabbix server" item1 "some text"

              If an entry contains special characters, they must be escaped with a backslash:

               "Zabbix server" item1 "sending \"some text\""

              A backslash itself must not be escaped:

               "Zabbix server" item1 "C:\Documents\My Folder"
               "Zabbix server" item1 C:\Documents

              In case the last character is a backslash double-quotes cannot be used:

               "Zabbix server" item1 C:\
               "Zabbix server" item1 "C:\Documents\"   <-- this will not work!

       -T, --with-timestamps

              This option can be only used with --input-file option.

              Each line of the input file must contain 4 whitespace delimited entries: <hostname> <key> <timestamp> <value>. Timestamp should be specified in Unix timestamp format. If target item has triggers referencing it, all
              timestamps must be in an increasing order, otherwise event calculation will not be correct.

              An example of a line of the input file:

               host1 item1 1429533600 sometext

              For more examples on the input file format please see option --input-file.
Comment by dimir [ 2015 Apr 21 ]

Let's try to list all the possible ways we could send data to server (tested with latest 2.0 (2.0.14)):

Entry in the input file Result in the database Comment
sometext
sometext
 
"some text"
some text
 
"sending \"some text\""
sending "some text"
 
"C:\My Documents"
C:\My Documents
 
C:\Documents
C:\Documents
 
C:\ 
C:\ 
 
"C:\"
error Warning: [line 1] 'Key value' required
some\ntext
some\ntext
 
"some\ntext"
some<LINEFEED>text
 
"some\ntext\n"
some<LINEFEED>text
linefeeds are always trimmed from the end
"\nsome\ntext\n"
<LINEFEED>some<LINEFEED>text
but not from the beginning
"\n\n"

linefeeds alone are trimmed though, which results in an empty value
some\\text
some\\text
 
"some\\text"
some\text
???
"some\\\text"
some\\text
three equals two
"some\\\\text"
some\\text
and four equals two
\ 
\ 
 
\\ 
\\ 
 
"
error Warning: [line 1] 'Key value' required
\"
\"
 
"\""
"
 
"\\""
error Warning: [line 1] 'Key value' required
"\\\""
\"
 

So based on above I came up with the following use cases:

  • an example of a line of the input file:
    host1 item1 sometext
  • entries containing a whitespace must be quoted with double-quotes (escaping a whitespace will not work):
    "Zabbix server" item1 "some text"
  • double quotes inside a double-quoted string must be escaped with a backslash:
    "Zabbix server" item1 "sending \"some text\""
  • a backslash in a non-quoted string does not need to be escaped:
    "Zabbix server" item1 C:\Documents
  • a backslash in a quoted string is tricky. In general, it's always safer to escape the backslashes in a quoted string. The reason is that both escaped and non-escaped backslashes are supported. This results in single and double backslashes being treated as one (both examples below will result in C:\My Documents):
    "Zabbix server" item1 "C:\My Documents"
    "Zabbix server" item1 "C:\\My Documents"

    single last character as a backslash is only supported in a non-quoted string:

    "Zabbix server" item1 C:\ 
    "Zabbix server" item1 "C:\"   <-- this will result in an error!

    so in order to have it in the quoted string, escape it:

    "Zabbix server" item1 "C:\\"

    in fact, it is safer to always escape the backslashes in the quoted string:

    "Zabbix server" item1 "C:\\My Documents\\"
  • escape sequences are not supported in non-quoted strings (this will result in some\ntext):
    "Zabbix server" item1 some\ntext

    in quoted strings linefeed escape sequences are supported, next line will result in each word on a separate line:

    "Zabbix server" item1 "some\ntext"

    however, linefeeds are trimmed from the end, so next line will produce the same result:

    "Zabbix server" item1 "some\ntext\n\n"

    linefeeds are not trimmed from the beginning:

    "Zabbix server" item1 "\n\nsome\ntext"

    however linefeeds alone will be completely removed (this will result in an empty string):

    "Zabbix server" item1 "\n\n"
Comment by dimir [ 2015 Apr 21 ]

So, if we separate the above into rules, use cases and notes for the man page:

The rules
  • quoted and non-quoted entries are supported
  • quoting character is double-quote
  • entries with whitespaces must be quoted
  • double-quote and backslash characters inside quoted entry must be escaped with a backslash
  • escaping is not supported in non-quoted entries
  • linefeed escape sequences (\n) are supported in quoted strings
  • linefeed escape sequences are trimmed from the end of an entry
Examples
entry result
sometext
sometext
"some text"
some text
"some \"text\""
some "text"
"C:\\"
C:\ 
C:\ 
C:\ 
some\ntext
some\ntext
"some\ttext"
some\ttext
"some\ntext"
some<LF>text
Notes

Single backslash characters can be used without escaping under certain conditions in quoted entries but please do not rely on it. Support for that might be removed in the future.

entry result
"C:\My Documents"
C:\My Documents
"C:\"
Warning: [line 1] 'Key value' required
"some\ntext\n\n"
some<LF>text
"\n\nsome\ntext"
<LF><LF>some<LF>text
"\n\n"

Comment by Martins Valkovskis [ 2015 Apr 22 ]

(3) Maybe worth changing "quoting character is double-quote" into "double-quote is the quoting character" (seems less ambiguous to me). Other than that, the above sounds good to me.

<dimir> Thanks, fixed in r53283. CLOSED

Comment by richlv [ 2015 Aug 28 ]

(4) "Entries with spaces must be quoted." - what about tab characters ? would it be safer to use "whitespace" here, like we do when describing the line format ?

<dimir> Good point, as TAB char actually is preserved. RESOLVED in r55448

<richlv> "Entries with whitespace must be quoted" would sound better - changed in r55450 (and i'm slightly sorry about the typo in the commit message )), still RESOLVED

<dimir> Nice :-D

CLOSED

Comment by richlv [ 2015 Aug 28 ]

(5) rules and notes currently duplicate this information :

  • Linefeed escape sequences are trimmed from the end of an entry.
  • Linefeed escape sequences (\n) are removed from the end of an entry.

i would suggest to remove the second entry

<dimir> RESOLVED in r55447

<richlv> currently it says :

Linefeed escape sequences (\n) are supported in quoted strings.
Linefeed escape sequences are trimmed from the end of an entry.

assuming that this is good, as "\n" is mentioned the first time we use "linefeed" in this manpage, but not in the second
CLOSED

Comment by richlv [ 2015 Aug 28 ]

(6) the last table in the "-i" section is a good thing to have in this issue or maybe in some wiki article, but it seems to document practice that is not really supported and suggested to be used - i would suggest to remove it and place the newline trimming examples in the first table

<dimir> RESOLVED in r55447

<richlv> looks like the decision was to remove those examples completely, as they are considered to document an unsupported behaviour - fine by me
CLOSED

Comment by richlv [ 2015 Sep 08 ]

(7) while we are at this, maybe it's a good idea to also mention the things referenced in (1) ? namely :

  • sending of up to 250 values in one connection
  • "All entries from an input file are sent in a sequential order top-down."
  • "If the target item has triggers referencing it, all timestamps in an input file must be in an increasing order, otherwise event calculation will not be correct." - and maybe an->the ?

if done so, i'd also suggest removing them from the specific manual page as we have manpages online and maintaining duplicate information has always ended up badly

<dimir> Fixed in r55652, r55654. Updated

https://www.zabbix.com/documentation/3.0/manual/concepts/sender

Rich, could you try to generate the web page from the manual to see how it works? The manual now has some fancy things like tables which might not work.

<richlv> indeed, the conversion process seems to get confused by the new bullet list. the table is there, but some fields have incorrect newlines and one seems to have incorrect contents. will investigate.

<dimir> Fixes man page format to be acceptable by man2html that comes with standard Debian installation. RESOLVED in r55690

<richlv> i'd suggest removing this part :

For example when specifying input file with 300 values an output might look something like this:

  1. zabbix_sender -z 127.0.0.1 -i /tmp/traptest.txt
    Info from server: "Processed 250 Failed 0 Total 250 Seconds spent 0.002668"
    Info from server: "Processed 50 Failed 0 Total 50 Seconds spent 0.000540"
    sent: 300; skipped: 0; total: 300

the sentence before that does explain the concept well enough, and any changes to the output will require manpage changes

<dimir> RESOLVED in r55788

<richlv> thanks, looks good to me
CLOSED

martins-v It doesn't look as though we have updated online copies of the manpages here:

In the meantime, users are complaining about the parts that were removed from the main manual, in ZBX-10026; also translators of the Russian manual for 2.4 are mentioning parts "lost" between 2.2 and 2.4 sender documentation.

REOPENED

<dimir> Correct me if I'm wrong, but it looks like 3.0 manual is up-to-date, the problem is with 2.2 and 2.4? As for removed information, do you suggest we keep 2 copies of the same info one in the manual and one in online manpages?

martins-v Apologies, it looks like indeed the problem is with 2.2. and 2.4 (not 3.0). I've fixed the corresponding links above.

Ignore my other comments. As long as we have the manpages online, we should not duplicate.

<dimir> OK, great, updated the 2.2 and 2.4 online manpages.

RESOLVED

martins-v Thanks, CLOSED.

Comment by richlv [ 2015 Sep 08 ]

(8) [D] when we are done, all the great work has to be mentioned in whatsnew

<dimir> Documentation updated:

martins-v I extended the sentences slightly, purely for consistent sentence structure. Please close the subissue, if satisfied.

<dimir> Looks great, thanks!

CLOSED

Comment by dimir [ 2015 Sep 30 ]

Recreated branch against 2.2, fixed in svn://svn.zabbix.com/branches/dev/ZBX-9347 (r55858).

Comment by dimir [ 2015 Oct 19 ]

Fixed in pre-2.2.11rc1 (r56246), pre-2.4.7rc1 (r56247), pre-3.0.0alpha4 (r56250).

Generated at Fri Mar 29 16:18:46 EET 2024 using Jira 9.12.4#9120004-sha1:625303b708afdb767e17cb2838290c41888e9ff0.