[Top][All Lists]
[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[10/18] argv_ref speedup: avoid extra string copying
|
From: |
Eric Blake |
|
Subject: |
[10/18] argv_ref speedup: avoid extra string copying |
|
Date: |
Wed, 16 Jan 2008 21:06:10 -0700 |
|
User-agent: |
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.9) Gecko/20071031 Thunderbird/2.0.0.9 Mnenhy/0.7.5.666 |
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Next in the series. I noticed that we were doing a lot of unnecessary
string copying, in two different places. First, for comments and quoted
strings, the input engine was copying text into the token stack, then the
argument collection copied from the token stack into the argument stack.
And since quoted strings tend to be the longest and most-frequently
handled input tokens, it was adding up. With this patch, strings are
copied directly into the argument stack (and future patches avoid even
that copy if a back-reference can be reused). Second, the format macro
was always mallocing a result, then copying that result to the obstack;
now, it tries to output the result in-place in the obstack when possible.
This patch has a side-effect that len(format(%c,0)) is now 1, not 0, but
as behavior of NUL (and for that matter, of format(%c)) is not documented,
I'm not too worried about the change. Later patches in the series will
add regression tests and documentation for NUL behavior in format, as I
make further cleanups.
2008-01-17 Eric Blake <address@hidden>
Stage 10: avoid extra copying of strings and comments.
When collecting tokens that are immune to further expansion, avoid
copying data from one obstack to another by outputting it into the
destination obstack to begin with. Also reduce copying done in
format builtin.
Memory impact: slight improvement, due to better obstack usage.
Speed impact: noticeable improvement, due less data copying.
* m4/gnulib-cache.m4: Import the intprops and vasnprintf-posix
modules.
* src/m4.h (includes): Use new gnulib modules.
(next_token): Adjust prototype.
(bad_argc): New prototype.
* src/format.c (arg_int, arg_long, arg_double): New helper
functions.
(ARG_INT, ARG_LONG, ARG_STR, ARG_DOUBLE): Track missing
arguments.
(format): Improve warnings, and avoid copying into obstack in the
common case.
* src/input.c (next_token): Add new parameter.
(lex_debug): Adjust caller.
* src/symtab.c (symtab_debug): Likewise.
* src/macro.c (expand_input, expand_token, expand_argument)
(collect_arguments): Likewise.
* src/output.c (m4_tmpname): Guarantee correct buffer size.
* src/builtin.c (builtin_init): Improve efficiency.
(bad_argc): Export.
(ntoa): Tighten buffer allocation.
* doc/m4.texinfo (Format): Test the improvements.
* NEWS: Document the improvement.
- --
Don't work too hard, make some time for fun as well!
Eric Blake address@hidden
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.5 (Cygwin)
Comment: Public key at home.comcast.net/~ericblake/eblake.gpg
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org
iD8DBQFHjtQy84KuGfSFAYARAlmOAJ961aRg3S/04EXWJ58KY3TzJnJMNACfcdSD
gh9Qhm/L9fD88BvFD0RgFE0=
=9dah
-----END PGP SIGNATURE-----
>From 782e3ac755755787d87a5057a6631329661be3ed Mon Sep 17 00:00:00 2001
From: Eric Blake <address@hidden>
Date: Wed, 16 Jan 2008 07:28:32 -0700
Subject: [PATCH] Stage 10: avoid extra copying of strings and comments.
* ltdl/m4/gnulib-cache.m4: Import intprops and vasnprintf-posix
modules.
* m4/m4private.h (m4__token_type): Adjust prototype.
* m4/input.c (m4__next_token): Support new parameter.
* m4/macro.c (m4_macro_expand_input, expand_token)
(expand_argument, collect_arguments): Adjust callers.
* modules/m4.c (ntoa): Tighten buffer size.
* m4/output.c (m4_tmpname): Guarantee no buffer overflow.
* modules/format.c (arg_int, arg_long, arg_double): New helper
functions, to detect overflow or unparsed characters.
(ARG_INT, ARG_LONG, ARG_STR, ARG_DOUBLE): Adjust to check for
missing or excess arguments.
(format): Likewise, and also output directly into obstack if there
is room.
* doc/m4.texinfo (History): Update for new year.
(Format): Test for new warnings.
Signed-off-by: Eric Blake <address@hidden>
---
ChangeLog | 26 +++++++
doc/m4.texinfo | 26 +++++--
ltdl/m4/gnulib-cache.m4 | 4 +-
m4/input.c | 99 +++++++++++++++++---------
m4/m4private.h | 4 +-
m4/macro.c | 46 ++++++++-----
m4/output.c | 21 ++++--
modules/format.c | 181 +++++++++++++++++++++++++++++++++++-----------
modules/m4.c | 9 ++-
9 files changed, 298 insertions(+), 118 deletions(-)
diff --git a/ChangeLog b/ChangeLog
index f7e861b..cc00596 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,29 @@
+2008-01-16 Eric Blake <address@hidden>
+
+ Stage 10: avoid extra copying of strings and comments.
+ When collecting tokens that are immune to further expansion, avoid
+ copying data from one obstack to another by outputting it into the
+ destination obstack to begin with. Also reduce copying done in
+ format builtin.
+ Memory impact: slight improvement, due to better obstack usage.
+ Speed impact: noticeable improvement, due less data copying.
+ * ltdl/m4/gnulib-cache.m4: Import intprops and vasnprintf-posix
+ modules.
+ * m4/m4private.h (m4__token_type): Adjust prototype.
+ * m4/input.c (m4__next_token): Support new parameter.
+ * m4/macro.c (m4_macro_expand_input, expand_token)
+ (expand_argument, collect_arguments): Adjust callers.
+ * modules/m4.c (ntoa): Tighten buffer size.
+ * m4/output.c (m4_tmpname): Guarantee no buffer overflow.
+ * modules/format.c (arg_int, arg_long, arg_double): New helper
+ functions, to detect overflow or unparsed characters.
+ (ARG_INT, ARG_LONG, ARG_STR, ARG_DOUBLE): Adjust to check for
+ missing or excess arguments.
+ (format): Likewise, and also output directly into obstack if there
+ is room.
+ * doc/m4.texinfo (History): Update for new year.
+ (Format): Test for new warnings.
+
2008-01-15 Eric Blake <address@hidden>
* TODO: Update with some newer URLs.
diff --git a/doc/m4.texinfo b/doc/m4.texinfo
index f56188f..df08093 100644
--- a/doc/m4.texinfo
+++ b/doc/m4.texinfo
@@ -45,7 +45,7 @@ This manual is for @acronym{GNU} M4 (version @value{VERSION},
@value{UPDATED}),
a package containing an implementation of the m4 macro language.
Copyright @copyright{} 1989, 1990, 1991, 1992, 1993, 1994, 1998, 1999,
-2000, 2001, 2004, 2005, 2006, 2007 Free Software Foundation, Inc.
+2000, 2001, 2004, 2005, 2006, 2007, 2008 Free Software Foundation, Inc.
@quotation
Permission is granted to copy, distribute and/or modify this document
@@ -410,9 +410,11 @@ addressed some long standing bugs in the venerable 1.4
release. Then in
2005, Gary V. Vaughan collected together the many patches to
@acronym{GNU} @code{m4} 1.4 that were floating around the net and
released 1.4.3 and 1.4.4. And in 2006, Eric Blake joined the team and
-prepared patches for the release of 1.4.5, 1.4.6, 1.4.7, and 1.4.8. The
-1.4.x series remains open for bug fixes, including releases 1.4.9,
-1.4.10, and 1.4.11 in 2007.
+prepared patches for the release of 1.4.5, 1.4.6, 1.4.7, and 1.4.8.
+More bug fixes were incorporated in 2007, with the releases of 1.4.9 and
+1.4.10. In 2008, Eric additionally rewrote the scanning engine to
+reduce recursive evaluation from quadratic to linear complexity for
+1.4.11. The 1.4.x branch remains open for bug fixes.
Meanwhile, development was underway for new features for @code{m4},
such as dynamic module loading and additional builtins, practically
@@ -6359,15 +6361,23 @@ example, @samp{%a} is supported even on platforms that
haven't yet
implemented C99 hexadecimal floating point output natively).
@c FIXME - format still needs some improvements.
-Unrecognized specifiers result in a warning. It is anticipated that a
-future release of @acronym{GNU} @code{m4} will support more specifiers,
-and give better warnings when various problems such as overflow are
-encountered. Likewise, escape sequences are not yet recognized.
+Warnings are issued for unrecognized specifiers, an improper number of
+arguments, or difficulty parsing an argument according to the format
+string (such as overflow or extra characters). It is anticipated that a
+future release of @acronym{GNU} @code{m4} will support more specifiers.
+Likewise, escape sequences are not yet recognized.
@example
format(`%p', `0')
@error{}m4:stdin:1: Warning: format: unrecognized specifier in `%p'
@result{}
+format(`%*d', `')
address@hidden:stdin:2: Warning: format: empty string treated as 0
address@hidden:stdin:2: Warning: format: too few arguments: 2 < 3
address@hidden
+format(`%.1f', `2a')
address@hidden:stdin:3: Warning: format: non-numeric argument `2a'
address@hidden
@end example
@node Arithmetic
diff --git a/ltdl/m4/gnulib-cache.m4 b/ltdl/m4/gnulib-cache.m4
index 6069529..3d01c5f 100644
--- a/ltdl/m4/gnulib-cache.m4
+++ b/ltdl/m4/gnulib-cache.m4
@@ -15,11 +15,11 @@
# Specification in the form of a command-line invocation:
-# gnulib-tool --import --dir=. --local-dir=local --lib=libgnu
--source-base=gnu --m4-base=ltdl/m4 --doc-base=doc --aux-dir=build-aux
--with-tests --libtool --macro-prefix=M4 assert autobuild avltree-oset
binary-io clean-temp cloexec close-stream closein config-h configmake dirname
error exit fdl fflush filenamecat flexmember fopen-safer free fseeko gendocs
gettext gnupload gpl-3.0 memmem mkstemp obstack progname quote regex
regexprops-generic sprintf-posix stdbool stdlib-safer strnlen strtol tempname
unlocked-io verror xalloc xalloc-die xprintf-posix xstrndup xvasprintf-posix
+# gnulib-tool --import --dir=. --local-dir=local --lib=libgnu
--source-base=gnu --m4-base=ltdl/m4 --doc-base=doc --aux-dir=build-aux
--with-tests --libtool --macro-prefix=M4 assert autobuild avltree-oset
binary-io clean-temp cloexec close-stream closein config-h configmake dirname
error exit fdl fflush filenamecat flexmember fopen-safer free fseeko gendocs
gettext gnupload gpl-3.0 intprops memmem mkstemp obstack progname quote regex
regexprops-generic sprintf-posix stdbool stdlib-safer strnlen strtol tempname
unlocked-io vasnprintf-posix verror xalloc xalloc-die xprintf-posix xstrndup
xvasprintf-posix
# Specification in the form of a few gnulib-tool.m4 macro invocations:
gl_LOCAL_DIR([local])
-gl_MODULES([assert autobuild avltree-oset binary-io clean-temp cloexec
close-stream closein config-h configmake dirname error exit fdl fflush
filenamecat flexmember fopen-safer free fseeko gendocs gettext gnupload gpl-3.0
memmem mkstemp obstack progname quote regex regexprops-generic sprintf-posix
stdbool stdlib-safer strnlen strtol tempname unlocked-io verror xalloc
xalloc-die xprintf-posix xstrndup xvasprintf-posix])
+gl_MODULES([assert autobuild avltree-oset binary-io clean-temp cloexec
close-stream closein config-h configmake dirname error exit fdl fflush
filenamecat flexmember fopen-safer free fseeko gendocs gettext gnupload gpl-3.0
intprops memmem mkstemp obstack progname quote regex regexprops-generic
sprintf-posix stdbool stdlib-safer strnlen strtol tempname unlocked-io
vasnprintf-posix verror xalloc xalloc-die xprintf-posix xstrndup
xvasprintf-posix])
gl_AVOID([])
gl_SOURCE_BASE([gnu])
gl_M4_BASE([ltdl/m4])
diff --git a/m4/input.c b/m4/input.c
index 26298fb..6dcaac0 100644
--- a/m4/input.c
+++ b/m4/input.c
@@ -1,6 +1,6 @@
/* GNU m4 -- A simple macro processor
- Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2006, 2007 Free Software
- Foundation, Inc.
+ Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2006, 2007, 2008
+ Free Software Foundation, Inc.
This file is part of GNU M4.
@@ -1144,17 +1144,19 @@ m4_input_exit (void)
/* Parse and return a single token from the input stream, built in
TOKEN. See m4__token_type for the valid return types, along with a
description of what TOKEN will contain. If LINE is not NULL, set
- *LINE to the line number where the token starts. Report errors
- (unterminated comments or strings) on behalf of CALLER, if
- non-NULL.
-
- The token text is collected on the obstack token_stack, which never
- contains more than one token text at a time. The storage pointed
- to by the fields in TOKEN is therefore subject to change the next
- time m4__next_token () is called. */
+ *LINE to the line number where the token starts. If OBS, expand
+ safe tokens (strings and comments) directly into OBS rather than in
+ a temporary staging area. Report errors (unterminated comments or
+ strings) on behalf of CALLER, if non-NULL.
+
+ If OBS is NULL or the token expansion is unknown, the token text is
+ collected on the obstack token_stack, which never contains more
+ than one token text at a time. The storage pointed to by the
+ fields in TOKEN is therefore subject to change the next time
+ m4__next_token () is called. */
m4__token_type
m4__next_token (m4 *context, m4_symbol_value *token, int *line,
- const char *caller)
+ m4_obstack *obs, const char *caller)
{
int ch;
int quote_level;
@@ -1162,6 +1164,11 @@ m4__next_token (m4 *context, m4_symbol_value *token, int
*line,
const char *file;
int dummy;
size_t len;
+ /* The obstack where token data is stored. Generally token_stack,
+ for tokens where argument collection might not use the literal
+ token. But for comments and strings, we can output directly into
+ the argument collection obstack OBS, if provided. */
+ m4_obstack *obs_safe = &token_stack;
assert (next == NULL);
if (!line)
@@ -1213,14 +1220,17 @@ m4__next_token (m4 *context, m4_symbol_value *token,
int *line,
}
else if (m4_has_syntax (M4SYNTAX, ch, M4_SYNTAX_ALPHA))
{
- obstack_1grow (&token_stack, ch);
- consume_syntax (context, &token_stack,
- M4_SYNTAX_ALPHA | M4_SYNTAX_NUM);
type = (m4_is_syntax_macro_escaped (M4SYNTAX)
? M4_TOKEN_STRING : M4_TOKEN_WORD);
+ if (type == M4_TOKEN_STRING && obs)
+ obs_safe = obs;
+ obstack_1grow (obs_safe, ch);
+ consume_syntax (context, obs_safe, M4_SYNTAX_ALPHA | M4_SYNTAX_NUM);
}
else if (m4_has_syntax (M4SYNTAX, ch, M4_SYNTAX_LQUOTE))
{ /* QUOTED STRING, SINGLE QUOTES
*/
+ if (obs)
+ obs_safe = obs;
quote_level = 1;
while (1)
{
@@ -1233,21 +1243,23 @@ m4__next_token (m4 *context, m4_symbol_value *token,
int *line,
{
if (--quote_level == 0)
break;
- obstack_1grow (&token_stack, ch);
+ obstack_1grow (obs_safe, ch);
}
else if (m4_has_syntax (M4SYNTAX, ch, M4_SYNTAX_LQUOTE))
{
quote_level++;
- obstack_1grow (&token_stack, ch);
+ obstack_1grow (obs_safe, ch);
}
else
- obstack_1grow (&token_stack, ch);
+ obstack_1grow (obs_safe, ch);
}
type = M4_TOKEN_STRING;
}
else if (!m4_is_syntax_single_quotes (M4SYNTAX)
&& MATCH (context, ch, context->syntax->lquote.string, true))
{ /* QUOTED STRING, LONGER QUOTES
*/
+ if (obs)
+ obs_safe = obs;
quote_level = 1;
while (1)
{
@@ -1259,28 +1271,30 @@ m4__next_token (m4 *context, m4_symbol_value *token,
int *line,
{
if (--quote_level == 0)
break;
- obstack_grow (&token_stack, context->syntax->rquote.string,
+ obstack_grow (obs_safe, context->syntax->rquote.string,
context->syntax->rquote.length);
}
else if (MATCH (context, ch, context->syntax->lquote.string, true))
{
quote_level++;
- obstack_grow (&token_stack, context->syntax->lquote.string,
+ obstack_grow (obs_safe, context->syntax->lquote.string,
context->syntax->lquote.length);
}
else
- obstack_1grow (&token_stack, ch);
+ obstack_1grow (obs_safe, ch);
}
type = M4_TOKEN_STRING;
}
else if (m4_has_syntax (M4SYNTAX, ch, M4_SYNTAX_BCOMM))
{ /* COMMENT, SHORT DELIM */
- obstack_1grow (&token_stack, ch);
+ if (obs && !m4_get_discard_comments_opt (context))
+ obs_safe = obs;
+ obstack_1grow (obs_safe, ch);
while ((ch = next_char (context, true)) != CHAR_EOF
&& !m4_has_syntax (M4SYNTAX, ch, M4_SYNTAX_ECOMM))
- obstack_1grow (&token_stack, ch);
+ obstack_1grow (obs_safe, ch);
if (ch != CHAR_EOF)
- obstack_1grow (&token_stack, ch);
+ obstack_1grow (obs_safe, ch);
else
m4_error_at_line (context, EXIT_FAILURE, 0, file, *line, caller,
_("end of file in comment"));
@@ -1290,13 +1304,15 @@ m4__next_token (m4 *context, m4_symbol_value *token,
int *line,
else if (!m4_is_syntax_single_comments (M4SYNTAX)
&& MATCH (context, ch, context->syntax->bcomm.string, true))
{ /* COMMENT, LONGER DELIM */
- obstack_grow (&token_stack, context->syntax->bcomm.string,
+ if (obs && !m4_get_discard_comments_opt (context))
+ obs_safe = obs;
+ obstack_grow (obs_safe, context->syntax->bcomm.string,
context->syntax->bcomm.length);
while ((ch = next_char (context, true)) != CHAR_EOF
&& !MATCH (context, ch, context->syntax->ecomm.string, true))
- obstack_1grow (&token_stack, ch);
+ obstack_1grow (obs_safe, ch);
if (ch != CHAR_EOF)
- obstack_grow (&token_stack, context->syntax->ecomm.string,
+ obstack_grow (obs_safe, context->syntax->ecomm.string,
context->syntax->ecomm.length);
else
m4_error_at_line (context, EXIT_FAILURE, 0, file, *line, caller,
@@ -1333,7 +1349,12 @@ m4__next_token (m4 *context, m4_symbol_value *token, int
*line,
(M4_SYNTAX_OTHER | M4_SYNTAX_NUM | M4_SYNTAX_DOLLAR
| M4_SYNTAX_LBRACE | M4_SYNTAX_RBRACE)))
{
- consume_syntax (context, &token_stack,
+ if (obs)
+ {
+ obs_safe = obs;
+ obstack_1grow (obs, ch);
+ }
+ consume_syntax (context, obs_safe,
(M4_SYNTAX_OTHER | M4_SYNTAX_NUM
| M4_SYNTAX_DOLLAR | M4_SYNTAX_LBRACE
| M4_SYNTAX_RBRACE));
@@ -1358,7 +1379,14 @@ m4__next_token (m4 *context, m4_symbol_value *token, int
*line,
if (m4_has_syntax (M4SYNTAX, ch,
(M4_SYNTAX_OTHER | M4_SYNTAX_NUM | M4_SYNTAX_DOLLAR
| M4_SYNTAX_LBRACE | M4_SYNTAX_RBRACE)))
- type = M4_TOKEN_STRING;
+ {
+ if (obs)
+ {
+ obs_safe = obs;
+ obstack_1grow (obs, ch);
+ }
+ type = M4_TOKEN_STRING;
+ }
else if (m4_has_syntax (M4SYNTAX, ch, M4_SYNTAX_SPACE))
type = M4_TOKEN_SPACE;
else
@@ -1366,12 +1394,17 @@ m4__next_token (m4 *context, m4_symbol_value *token,
int *line,
}
} while (type == M4_TOKEN_NONE);
- len = obstack_object_size (&token_stack);
- obstack_1grow (&token_stack, '\0');
+ if (obs_safe != obs)
+ {
+ len = obstack_object_size (&token_stack);
+ obstack_1grow (&token_stack, '\0');
- m4_set_symbol_value_text (token, obstack_finish (&token_stack), len,
- m4__quote_age (M4SYNTAX));
- VALUE_MAX_ARGS (token) = -1;
+ m4_set_symbol_value_text (token, obstack_finish (&token_stack), len,
+ m4__quote_age (M4SYNTAX));
+ }
+ else
+ assert (type == M4_TOKEN_STRING);
+ VALUE_MAX_ARGS (token) = -1;
#ifdef DEBUG_INPUT
m4_print_token ("next_token", type, token);
diff --git a/m4/m4private.h b/m4/m4private.h
index 272069e..630a9b7 100644
--- a/m4/m4private.h
+++ b/m4/m4private.h
@@ -1,7 +1,7 @@
/* GNU m4 -- A simple macro processor
Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 1998, 1999, 2004, 2005,
- 2006, 2007 Free Software Foundation, Inc.
+ 2006, 2007, 2008 Free Software Foundation, Inc.
This file is part of GNU M4.
@@ -453,7 +453,7 @@ typedef enum {
extern bool m4__push_symbol (m4 *, m4_symbol_value *, size_t);
extern m4__token_type m4__next_token (m4 *, m4_symbol_value *, int *,
- const char *);
+ m4_obstack *, const char *);
extern bool m4__next_token_is_open (m4 *);
/* Fast macro versions of macro argv accessor functions,
diff --git a/m4/macro.c b/m4/macro.c
index 15ec4d9..9963409 100644
--- a/m4/macro.c
+++ b/m4/macro.c
@@ -1,6 +1,6 @@
/* GNU m4 -- A simple macro processor
- Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2001, 2006, 2007
- Free Software Foundation, Inc.
+ Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2001, 2006, 2007,
+ 2008 Free Software Foundation, Inc.
This file is part of GNU M4.
@@ -183,9 +183,9 @@ m4_macro_expand_input (m4 *context)
m4_set_symbol_value_text (&empty_symbol, "", 0, 0);
VALUE_MAX_ARGS (&empty_symbol) = -1;
- while ((type = m4__next_token (context, &token, &line, NULL))
+ while ((type = m4__next_token (context, &token, &line, NULL, NULL))
!= M4_TOKEN_EOF)
- expand_token (context, (m4_obstack *) NULL, type, &token, line, true);
+ expand_token (context, NULL, type, &token, line, true);
}
@@ -221,8 +221,12 @@ expand_token (m4 *context, m4_obstack *obs, m4__token_type
type,
detects any change in delimiters). This is also returned for
sequences of benign characters, such as digits. But if other
text is already present, multi-character delimiters could be
- formed by concatenation, so use a conservative heuristic. */
+ formed by concatenation, so use a conservative heuristic. If
+ obstack was provided, the string was already expanded into it
+ during m4__next_token. */
result = first || m4__safe_quotes (M4SYNTAX);
+ if (obs)
+ return result;
break;
case M4_TOKEN_OPEN:
@@ -301,17 +305,22 @@ expand_argument (m4 *context, m4_obstack *obs,
m4_symbol_value *argp,
unsigned int age = m4__quote_age (M4SYNTAX);
bool first = true;
- argp->type = M4_SYMBOL_VOID;
+ memset (argp, '\0', sizeof *argp);
+ VALUE_MAX_ARGS (argp) = -1;
/* Skip leading white space. */
do
{
- type = m4__next_token (context, &token, NULL, caller);
+ type = m4__next_token (context, &token, NULL, obs, caller);
}
while (type == M4_TOKEN_SPACE);
while (1)
{
+ if (VALUE_MIN_ARGS (argp) < VALUE_MIN_ARGS (&token))
+ VALUE_MIN_ARGS (argp) = VALUE_MIN_ARGS (&token);
+ if (VALUE_MAX_ARGS (&token) < VALUE_MAX_ARGS (argp))
+ VALUE_MAX_ARGS (argp) = VALUE_MAX_ARGS (&token);
switch (type)
{ /* TOKSW */
case M4_TOKEN_COMMA:
@@ -367,7 +376,7 @@ expand_argument (m4 *context, m4_obstack *obs,
m4_symbol_value *argp,
if (argp->type != M4_SYMBOL_VOID || obstack_object_size (obs))
first = false;
- type = m4__next_token (context, &token, NULL, caller);
+ type = m4__next_token (context, &token, NULL, obs, caller);
}
}
@@ -553,18 +562,21 @@ collect_arguments (m4 *context, const char *name, size_t
len,
if (m4__next_token_is_open (context))
{
- m4__next_token (context, &token, NULL, name); /* gobble parenthesis */
+ /* Gobble parenthesis, then collect arguments. */
+ m4__next_token (context, &token, NULL, NULL, name);
do
{
- more_args = expand_argument (context, arguments, &token, name);
+ tokenp = (m4_symbol_value *) obstack_alloc (arguments,
+ sizeof *tokenp);
+ more_args = expand_argument (context, arguments, tokenp, name);
- if ((m4_is_symbol_value_text (&token)
- && !m4_get_symbol_value_len (&token))
- || (!groks_macro_args && m4_is_symbol_value_func (&token)))
- tokenp = &empty_symbol;
- else
- tokenp = (m4_symbol_value *) obstack_copy (arguments, &token,
- sizeof *tokenp);
+ if ((m4_is_symbol_value_text (tokenp)
+ && !m4_get_symbol_value_len (tokenp))
+ || (!groks_macro_args && m4_is_symbol_value_func (tokenp)))
+ {
+ obstack_free (arguments, tokenp);
+ tokenp = &empty_symbol;
+ }
obstack_ptr_grow (argv_stack, tokenp);
args.arraylen++;
args.argc++;
diff --git a/m4/output.c b/m4/output.c
index ab46994..f745efe 100644
--- a/m4/output.c
+++ b/m4/output.c
@@ -1,6 +1,6 @@
/* GNU m4 -- A simple macro processor
- Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 1998, 2002, 2004, 2006,
- 2007 Free Software Foundation, Inc.
+ Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 1998, 2002, 2004,
+ 2006, 2007, 2008 Free Software Foundation, Inc.
This file is part of GNU M4.
@@ -190,15 +190,20 @@ static const char *
m4_tmpname (int divnum)
{
static char *buffer;
- static char *tail;
+ static size_t offset;
if (buffer == NULL)
{
- tail = xasprintf ("%s/m4-%d", output_temp_dir->dir_name, INT_MAX);
- buffer = obstack_copy0 (&diversion_storage, tail, strlen (tail));
- free (tail);
- tail = strrchr (buffer, '-') + 1;
+ obstack_grow (&diversion_storage, output_temp_dir->dir_name,
+ strlen (output_temp_dir->dir_name));
+ obstack_1grow (&diversion_storage, '/');
+ obstack_1grow (&diversion_storage, 'm');
+ obstack_1grow (&diversion_storage, '4');
+ obstack_1grow (&diversion_storage, '-');
+ offset = obstack_object_size (&diversion_storage);
+ buffer = obstack_alloc (&diversion_storage, INT_BUFSIZE_BOUND (divnum));
}
- sprintf (tail, "%d", divnum);
+ if (snprintf (&buffer[offset], INT_BUFSIZE_BOUND (divnum), "%d", divnum) < 0)
+ abort ();
return buffer;
}
diff --git a/modules/format.c b/modules/format.c
index ced3924..fd086c8 100644
--- a/modules/format.c
+++ b/modules/format.c
@@ -1,6 +1,6 @@
/* GNU m4 -- A simple macro processor
- Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2001, 2006, 2007
- Free Software Foundation, Inc.
+ Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2001, 2006, 2007,
+ 2008 Free Software Foundation, Inc.
This file is part of GNU M4.
@@ -20,27 +20,96 @@
/* printf like formatting for m4. */
-#include "xvasprintf.h"
+#include "vasnprintf.h"
/* Simple varargs substitute. We assume int and unsigned int are the
- same size; likewise for long and unsigned long.
+ same size; likewise for long and unsigned long. We do not yet
+ handle long double or long long. */
- TODO - warn if we use these because too many % specifiers were used in
- relation to number of arguments passed.
- TODO - use xstrtoimax, not atoi, to catch overflow, non-numeric
- arguments, etc. */
+/* Parse STR as an integer, reporting warnings on behalf of ME. */
+static int
+arg_int (struct m4 *context, const char *me, const char *str)
+{
+ char *endp;
+ long value;
+
+ /* TODO - also allow parsing `'a' or `"a' which results in the
+ numeric value of 'a', as in printf(1). */
+ if (*str == '\0')
+ {
+ m4_warn (context, 0, me, _("empty string treated as 0"));
+ return 0;
+ }
+ errno = 0;
+ value = strtol (str, &endp, 10);
+ if (*endp != '\0')
+ m4_warn (context, 0, me, _("non-numeric argument `%s'"), str);
+ else if (isspace (to_uchar (*str)))
+ m4_warn (context, 0, me, _("leading whitespace ignored"));
+ else if (errno == ERANGE || (int) value != value)
+ m4_warn (context, 0, me, _("numeric overflow detected"));
+ return value;
+}
-#define ARG_INT(i, argc, argv) \
- ((argc <= i++) ? 0 : atoi (M4ARG (i - 1)))
+/* Parse STR as a long, reporting warnings on behalf of ME. */
+static long
+arg_long (struct m4 *context, const char *me, const char *str)
+{
+ char *endp;
+ long value;
-#define ARG_LONG(i, argc, argv) \
- ((argc <= i++) ? 0L : atol (M4ARG (i - 1)))
+ /* TODO - also allow parsing `'a' or `"a' which results in the
+ numeric value of 'a', as in printf(1). */
+ if (*str == '\0')
+ {
+ m4_warn (context, 0, me, _("empty string treated as 0"));
+ return 0L;
+ }
+ errno = 0;
+ value = strtol (str, &endp, 10);
+ if (*endp != '\0')
+ m4_warn (context, 0, me, _("non-numeric argument `%s'"), str);
+ else if (isspace (to_uchar (*str)))
+ m4_warn (context, 0, me, _("leading whitespace ignored"));
+ else if (errno == ERANGE)
+ m4_warn (context, 0, me, _("numeric overflow detected"));
+ return value;
+}
-#define ARG_STR(i, argc, argv) \
- ((argc <= i++) ? "" : M4ARG (i - 1))
+/* Parse STR as a double, reporting warnings on behalf of ME. */
+static double
+arg_double (struct m4 *context, const char *me, const char *str)
+{
+ char *endp;
+ double value;
-#define ARG_DOUBLE(i, argc, argv) \
- ((argc <= i++) ? 0.0 : atof (M4ARG (i - 1)))
+ if (*str == '\0')
+ {
+ m4_warn (context, 0, me, _("empty string treated as 0"));
+ return 0.0;
+ }
+ errno = 0;
+ value = strtod (str, &endp);
+ if (*endp != '\0')
+ m4_warn (context, 0, me, _("non-numeric argument `%s'"), str);
+ else if (isspace (to_uchar (*str)))
+ m4_warn (context, 0, me, _("leading whitespace ignored"));
+ else if (errno == ERANGE)
+ m4_warn (context, 0, me, _("numeric overflow detected"));
+ return value;
+}
+
+#define ARG_INT(i, argc, argv) \
+ ((argc <= ++i) ? 0 : arg_int (context, me, M4ARG (i)))
+
+#define ARG_LONG(i, argc, argv) \
+ ((argc <= ++i) ? 0L : arg_long (context, me, M4ARG (i)))
+
+#define ARG_STR(i, argc, argv) \
+ ((argc <= ++i) ? "" : M4ARG (i))
+
+#define ARG_DOUBLE(i, argc, argv) \
+ ((argc <= ++i) ? 0.0 : arg_double (context, me, M4ARG (i)))
/* The main formatting function. Output is placed on the obstack OBS,
@@ -52,30 +121,31 @@
static void
format (m4 *context, m4_obstack *obs, int argc, m4_macro_args *argv)
{
- const char *name = M4ARG (0); /* Macro name. */
+ const char *me = M4ARG (0); /* Macro name. */
const char *f; /* Format control string. */
const char *fmt; /* Position within f. */
char fstart[] = "%'+- 0#*.*hhd"; /* Current format spec. */
char *p; /* Position within fstart. */
unsigned char c; /* A simple character. */
- int index = 1; /* Index within argc used so far. */
+ int index = 0; /* Index within argc used so far. */
+ bool valid_format = true; /* True if entire format string ok. */
/* Flags. */
- char flags; /* flags to use in fstart */
+ char flags; /* Flags to use in fstart. */
enum {
- THOUSANDS = 0x01, /* ' */
- PLUS = 0x02, /* + */
- MINUS = 0x04, /* - */
- SPACE = 0x08, /* */
- ZERO = 0x10, /* 0 */
- ALT = 0x20, /* # */
- DONE = 0x40 /* no more flags */
+ THOUSANDS = 0x01, /* '\''. */
+ PLUS = 0x02, /* '+'. */
+ MINUS = 0x04, /* '-'. */
+ SPACE = 0x08, /* ' '. */
+ ZERO = 0x10, /* '0'. */
+ ALT = 0x20, /* '#'. */
+ DONE = 0x40 /* No more flags. */
};
/* Precision specifiers. */
- int width; /* minimum field width */
- int prec; /* precision */
- char lflag; /* long flag */
+ int width; /* Minimum field width. */
+ int prec; /* Precision. */
+ char lflag; /* Long flag. */
/* Specifiers we are willing to accept. ok['x'] implies %x is ok.
Various modifiers reduce the set, in order to avoid undefined
@@ -83,17 +153,23 @@ format (m4 *context, m4_obstack *obs, int argc,
m4_macro_args *argv)
char ok[128];
/* Buffer and stuff. */
- char *str; /* malloc'd buffer of formatted text */
+ char *base; /* Current position in obs. */
+ size_t len; /* Length of formatted text. */
+ char *str; /* Malloc'd buffer of formatted text. */
enum {CHAR, INT, LONG, DOUBLE, STR} datatype;
f = fmt = ARG_STR (index, argc, argv);
memset (ok, 0, sizeof ok);
- for (;;)
+ while (true)
{
while ((c = *fmt++) != '%')
{
if (c == '\0')
- return;
+ {
+ if (valid_format)
+ m4_bad_argc (context, argc, me, index, index, true);
+ return;
+ }
obstack_1grow (obs, c);
}
@@ -229,7 +305,8 @@ format (m4 *context, m4_obstack *obs, int argc,
m4_macro_args *argv)
c = *fmt++;
if (c > sizeof ok || !ok[c])
{
- m4_warn (context, 0, name, _("unrecognized specifier in `%s'"), f);
+ m4_warn (context, 0, me, _("unrecognized specifier in `%s'"), f);
+ valid_format = false;
if (c == '\0')
fmt--;
continue;
@@ -272,40 +349,56 @@ format (m4 *context, m4_obstack *obs, int argc,
m4_macro_args *argv)
}
*p++ = c;
*p = '\0';
+ base = obstack_next_free (obs);
+ len = obstack_room (obs);
switch (datatype)
{
case CHAR:
- str = xasprintf (fstart, width, ARG_INT (index, argc, argv));
+ str = asnprintf (base, &len, fstart, width,
+ ARG_INT (index, argc, argv));
break;
case INT:
- str = xasprintf (fstart, width, prec, ARG_INT (index, argc, argv));
+ str = asnprintf (base, &len, fstart, width, prec,
+ ARG_INT (index, argc, argv));
break;
case LONG:
- str = xasprintf (fstart, width, prec, ARG_LONG (index, argc, argv));
+ str = asnprintf (base, &len, fstart, width, prec,
+ ARG_LONG (index, argc, argv));
break;
case DOUBLE:
- str = xasprintf (fstart, width, prec,
+ str = asnprintf (base, &len, fstart, width, prec,
ARG_DOUBLE (index, argc, argv));
break;
case STR:
- str = xasprintf (fstart, width, prec, ARG_STR (index, argc, argv));
+ str = asnprintf (base, &len, fstart, width, prec,
+ ARG_STR (index, argc, argv));
break;
default:
abort ();
}
- /* NULL was returned on failure, such as invalid format string. For
- now, just silently ignore that bad specifier. */
if (str == NULL)
- continue;
-
- obstack_grow (obs, str, strlen (str));
- free (str);
+ /* NULL is unexpected (EILSEQ and EINVAL are not possible
+ based on our construction of fstart, leaving only ENOMEM,
+ which should always be fatal). */
+ m4_error (context, EXIT_FAILURE, errno, me,
+ _("unable to format output for `%s'"), f);
+ else if (str == base)
+ /* The output was already computed in place, but we need to
+ account for its size. */
+ obstack_blank_fast (obs, len);
+ else
+ {
+ /* The output exceeded available obstack space, copy the
+ allocated string. */
+ obstack_grow (obs, str, len);
+ free (str);
+ }
}
}
diff --git a/modules/m4.c b/modules/m4.c
index 6a2d1b6..4b1416e 100644
--- a/modules/m4.c
+++ b/modules/m4.c
@@ -1,5 +1,5 @@
/* GNU m4 -- A simple macro processor
- Copyright (C) 2000, 2002, 2003, 2004, 2006, 2007 Free Software
+ Copyright (C) 2000, 2002, 2003, 2004, 2006, 2007, 2008 Free Software
Foundation, Inc.
This file is part of GNU M4.
@@ -1128,7 +1128,7 @@ M4BUILTIN_HANDLER (translit)
/* The function ntoa () converts VALUE to a signed ascii representation in
- radix RADIX. */
+ radix RADIX. Radix must be between 2 and 36, inclusive. */
static const char *
ntoa (number value, int radix)
{
@@ -1137,7 +1137,8 @@ ntoa (number value, int radix)
bool negative;
unumber uvalue;
- static char str[256];
+ /* Sized for radix 2, plus sign and trailing NUL. */
+ static char str[sizeof value * CHAR_BIT + 2];
char *s = &str[sizeof str];
*--s = '\0';
@@ -1166,7 +1167,7 @@ ntoa (number value, int radix)
}
static void
-numb_obstack(m4_obstack *obs, number value, int radix, int min)
+numb_obstack (m4_obstack *obs, number value, int radix, int min)
{
const char *s;
size_t len;
--
1.5.3.8
>From 5d61bd60454bca489dc2f4eb4ee0d9eba4f1f425 Mon Sep 17 00:00:00 2001
From: Eric Blake <address@hidden>
Date: Fri, 26 Oct 2007 10:45:51 -0600
Subject: [PATCH] Stage 10: avoid extra copying of strings and comments.
* m4/gnulib-cache.m4: Import the intprops and vasnprintf-posix
modules.
* src/m4.h (includes): Use new gnulib modules.
(next_token): Adjust prototype.
(bad_argc): New prototype.
* src/format.c (arg_int, arg_long, arg_double): New helper
functions.
(ARG_INT, ARG_LONG, ARG_STR, ARG_DOUBLE): Track missing
arguments.
(format): Improve warnings, and avoid copying into obstack in the
common case.
* src/input.c (next_token): Add new parameter.
(lex_debug): Adjust caller.
* src/symtab.c (symtab_debug): Likewise.
* src/macro.c (expand_input, expand_token, expand_argument)
(collect_arguments): Likewise.
* src/output.c (m4_tmpname): Guarantee correct buffer size.
* src/builtin.c (builtin_init): Improve efficiency.
(bad_argc): Export.
(ntoa): Tighten buffer allocation.
* doc/m4.texinfo (Format): Test the improvements.
* NEWS: Document the improvement.
(cherry picked from commit 622fc8cb2cb6ce0fc7391a6414bb0aaffeec6fc0)
Signed-off-by: Eric Blake <address@hidden>
---
ChangeLog | 32 ++++++++++
NEWS | 2 +
doc/m4.texinfo | 16 ++++-
m4/gnulib-cache.m4 | 4 +-
src/builtin.c | 10 +--
src/format.c | 173 ++++++++++++++++++++++++++++++++++++++++------------
src/input.c | 90 ++++++++++++++++-----------
src/m4.h | 9 ++-
src/macro.c | 32 ++++++----
src/output.c | 18 +++--
src/symtab.c | 6 +-
11 files changed, 278 insertions(+), 114 deletions(-)
diff --git a/ChangeLog b/ChangeLog
index 4a20b00..5ad26e3 100644
--- a/ChangeLog
+++ b/ChangeLog
@@ -1,3 +1,35 @@
+2008-01-17 Eric Blake <address@hidden>
+
+ Stage 10: avoid extra copying of strings and comments.
+ When collecting tokens that are immune to further expansion, avoid
+ copying data from one obstack to another by outputting it into the
+ destination obstack to begin with. Also reduce copying done in
+ format builtin.
+ Memory impact: slight improvement, due to better obstack usage.
+ Speed impact: noticeable improvement, due less data copying.
+ * m4/gnulib-cache.m4: Import the intprops and vasnprintf-posix
+ modules.
+ * src/m4.h (includes): Use new gnulib modules.
+ (next_token): Adjust prototype.
+ (bad_argc): New prototype.
+ * src/format.c (arg_int, arg_long, arg_double): New helper
+ functions.
+ (ARG_INT, ARG_LONG, ARG_STR, ARG_DOUBLE): Track missing
+ arguments.
+ (format): Improve warnings, and avoid copying into obstack in the
+ common case.
+ * src/input.c (next_token): Add new parameter.
+ (lex_debug): Adjust caller.
+ * src/symtab.c (symtab_debug): Likewise.
+ * src/macro.c (expand_input, expand_token, expand_argument)
+ (collect_arguments): Likewise.
+ * src/output.c (m4_tmpname): Guarantee correct buffer size.
+ * src/builtin.c (builtin_init): Improve efficiency.
+ (bad_argc): Export.
+ (ntoa): Tighten buffer allocation.
+ * doc/m4.texinfo (Format): Test the improvements.
+ * NEWS: Document the improvement.
+
2008-01-14 Eric Blake <address@hidden>
Fix --warn-macro-sequence regression from 2007-11-29.
diff --git a/NEWS b/NEWS
index d7a3bae..31a41c2 100644
--- a/NEWS
+++ b/NEWS
@@ -25,6 +25,8 @@ Version 1.4.11 - ?? ??? 2008, by ???? (git version 1.4.10a-*)
systems with a brain-dead quadratic strstr(3).
* Enhance the `regexp' and `patsubst' builtins to cache frequently used
regular expressions, which speeds up typical Autoconf usage.
+* Enhance the `format' builtin to warn for more suspicious usages, such as
+ missing arguments or problems parsing according to the format string.
* Memory usage is greatly reduced in recursive macros.
* A number of portability improvements inherited from gnulib.
diff --git a/doc/m4.texinfo b/doc/m4.texinfo
index b24cc90..bcdb99b 100644
--- a/doc/m4.texinfo
+++ b/doc/m4.texinfo
@@ -5330,15 +5330,23 @@ see the C Library Manual, or the @acronym{POSIX}
specification (for
example, @samp{%a} is supported even on platforms that haven't yet
implemented C99 hexadecimal floating point output natively).
-Unrecognized specifiers result in a warning. It is anticipated that a
-future release of @acronym{GNU} @code{m4} will support more specifiers,
-and give better warnings when various problems such as overflow are
-encountered. Likewise, escape sequences are not yet recognized.
+Warnings are issued for unrecognized specifiers, an improper number of
+arguments, or difficulty parsing an argument according to the format
+string (such as overflow or extra characters). It is anticipated that a
+future release of @acronym{GNU} @code{m4} will support more specifiers.
+Likewise, escape sequences are not yet recognized.
@example
format(`%p', `0')
@error{}m4:stdin:1: Warning: format: unrecognized specifier in `%p'
@result{}
+format(`%*d', `')
address@hidden:stdin:2: Warning: format: empty string treated as 0
address@hidden:stdin:2: Warning: format: too few arguments: 2 < 3
address@hidden
+format(`%.1f', `2a')
address@hidden:stdin:3: Warning: format: non-numeric argument `2a'
address@hidden
@end example
@node Arithmetic
diff --git a/m4/gnulib-cache.m4 b/m4/gnulib-cache.m4
index 3112f91..0055a69 100644
--- a/m4/gnulib-cache.m4
+++ b/m4/gnulib-cache.m4
@@ -15,11 +15,11 @@
# Specification in the form of a command-line invocation:
-# gnulib-tool --import --dir=. --local-dir=local --lib=libm4
--source-base=lib --m4-base=m4 --doc-base=doc --aux-dir=build-aux --with-tests
--no-libtool --macro-prefix=M4 assert avltree-oset binary-io clean-temp cloexec
close-stream closein config-h error fdl fflush flexmember fopen-safer free
fseeko gendocs getopt gnupload gpl-3.0 memmem mkstemp obstack quote regex
stdbool stdint stdlib-safer strtol unlocked-io verror version-etc
version-etc-fsf xalloc xprintf xvasprintf-posix
+# gnulib-tool --import --dir=. --local-dir=local --lib=libm4
--source-base=lib --m4-base=m4 --doc-base=doc --aux-dir=build-aux --with-tests
--no-libtool --macro-prefix=M4 assert avltree-oset binary-io clean-temp cloexec
close-stream closein config-h error fdl fflush flexmember fopen-safer free
fseeko gendocs getopt gnupload gpl-3.0 intprops memmem mkstemp obstack quote
regex stdbool stdint stdlib-safer strtol unlocked-io vasnprintf-posix verror
version-etc version-etc-fsf xalloc xprintf xvasprintf-posix
# Specification in the form of a few gnulib-tool.m4 macro invocations:
gl_LOCAL_DIR([local])
-gl_MODULES([assert avltree-oset binary-io clean-temp cloexec close-stream
closein config-h error fdl fflush flexmember fopen-safer free fseeko gendocs
getopt gnupload gpl-3.0 memmem mkstemp obstack quote regex stdbool stdint
stdlib-safer strtol unlocked-io verror version-etc version-etc-fsf xalloc
xprintf xvasprintf-posix])
+gl_MODULES([assert avltree-oset binary-io clean-temp cloexec close-stream
closein config-h error fdl fflush flexmember fopen-safer free fseeko gendocs
getopt gnupload gpl-3.0 intprops memmem mkstemp obstack quote regex stdbool
stdint stdlib-safer strtol unlocked-io vasnprintf-posix verror version-etc
version-etc-fsf xalloc xprintf xvasprintf-posix])
gl_AVOID([])
gl_SOURCE_BASE([lib])
gl_M4_BASE([m4])
diff --git a/src/builtin.c b/src/builtin.c
index e873061..007ca55 100644
--- a/src/builtin.c
+++ b/src/builtin.c
@@ -465,12 +465,9 @@ builtin_init (void)
for (bp = &builtin_tab[0]; bp->name != NULL; bp++)
if (!no_gnu_extensions || !bp->gnu_extension)
{
- size_t len = strlen (bp->name);
if (prefix_all_builtins)
{
- string = xcharalloc (len + 4);
- strcpy (string, "m4_");
- strcat (string, bp->name);
+ string = xasprintf ("m4_%s", bp->name);
define_builtin (string, bp, SYMBOL_INSERT);
free (string);
}
@@ -502,7 +499,7 @@ builtin_init (void)
| Return true if there are not enough arguments. |
`------------------------------------------------------------------*/
-static bool
+bool
bad_argc (const char *name, int argc, unsigned int min, unsigned int max)
{
if (argc - 1 < min)
@@ -561,7 +558,8 @@ ntoa (int32_t value, int radix)
{
bool negative;
uint32_t uvalue;
- static char str[256];
+ /* Sized for radix 2, plus sign and trailing NUL. */
+ static char str[sizeof (value) * CHAR_BIT + 2];
char *s = &str[sizeof str];
*--s = '\0';
diff --git a/src/format.c b/src/format.c
index 20b3e28..9c9508d 100644
--- a/src/format.c
+++ b/src/format.c
@@ -1,6 +1,6 @@
/* GNU m4 -- A simple macro processor
- Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2006, 2007
+ Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2006, 2007, 2008
Free Software Foundation, Inc.
This file is part of GNU M4.
@@ -22,22 +22,95 @@
/* printf like formatting for m4. */
#include "m4.h"
-#include "xvasprintf.h"
/* Simple varargs substitute. We assume int and unsigned int are the
- same size; likewise for long and unsigned long. */
+ same size; likewise for long and unsigned long. We do not yet
+ handle long double or long long. */
+
+/* Parse STR as an integer, reporting warnings on behalf of ME. */
+static int
+arg_int (const char *me, const char *str)
+{
+ char *endp;
+ long value;
+
+ /* TODO - also allow parsing `'a' or `"a' which results in the
+ numeric value of 'a', as in printf(1). */
+ if (*str == '\0')
+ {
+ m4_warn (0, me, _("empty string treated as 0"));
+ return 0;
+ }
+ errno = 0;
+ value = strtol (str, &endp, 10);
+ if (*endp != '\0')
+ m4_warn (0, me, _("non-numeric argument `%s'"), str);
+ else if (isspace (to_uchar (*str)))
+ m4_warn (0, me, _("leading whitespace ignored"));
+ else if (errno == ERANGE || (int) value != value)
+ m4_warn (0, me, _("numeric overflow detected"));
+ return value;
+}
+
+/* Parse STR as a long, reporting warnings on behalf of ME. */
+static long
+arg_long (const char *me, const char *str)
+{
+ char *endp;
+ long value;
+
+ /* TODO - also allow parsing `'a' or `"a' which results in the
+ numeric value of 'a', as in printf(1). */
+ if (*str == '\0')
+ {
+ m4_warn (0, me, _("empty string treated as 0"));
+ return 0L;
+ }
+ errno = 0;
+ value = strtol (str, &endp, 10);
+ if (*endp != '\0')
+ m4_warn (0, me, _("non-numeric argument `%s'"), str);
+ else if (isspace (to_uchar (*str)))
+ m4_warn (0, me, _("leading whitespace ignored"));
+ else if (errno == ERANGE)
+ m4_warn (0, me, _("numeric overflow detected"));
+ return value;
+}
+
+/* Parse STR as a double, reporting warnings on behalf of ME. */
+static double
+arg_double (const char *me, const char *str)
+{
+ char *endp;
+ double value;
+
+ if (*str == '\0')
+ {
+ m4_warn (0, me, _("empty string treated as 0"));
+ return 0.0;
+ }
+ errno = 0;
+ value = strtod (str, &endp);
+ if (*endp != '\0')
+ m4_warn (0, me, _("non-numeric argument `%s'"), str);
+ else if (isspace (to_uchar (*str)))
+ m4_warn (0, me, _("leading whitespace ignored"));
+ else if (errno == ERANGE)
+ m4_warn (0, me, _("numeric overflow detected"));
+ return value;
+}
#define ARG_INT(i, argc, argv) \
- ((i == argc) ? 0 : atoi (arg_text (argv, i++)))
+ ((argc <= ++i) ? 0 : arg_int (me, arg_text (argv, i)))
#define ARG_LONG(i, argc, argv)
\
- ((i == argc) ? 0L : atol (arg_text (argv, i++)))
+ ((argc <= ++i) ? 0L : arg_long (me, arg_text (argv, i)))
#define ARG_STR(i, argc, argv) \
- ((i == argc) ? "" : arg_text (argv, i++))
+ ((argc <= ++i) ? "" : arg_text (argv, i))
#define ARG_DOUBLE(i, argc, argv) \
- ((i == argc) ? 0.0 : atof (arg_text (argv, i++)))
+ ((argc <= ++i) ? 0.0 : arg_double (me, arg_text (argv, i)))
/*------------------------------------------------------------------.
@@ -52,29 +125,30 @@ void
format (struct obstack *obs, int argc, macro_arguments *argv)
{
const char *me = arg_text (argv, 0);
- const char *f; /* format control string */
- const char *fmt; /* position within f */
- char fstart[] = "%'+- 0#*.*hhd"; /* current format spec */
- char *p; /* position within fstart */
- unsigned char c; /* a simple character */
- int index = 1; /* index within argc used so far */
+ const char *f; /* Format control string. */
+ const char *fmt; /* Position within f. */
+ char fstart[] = "%'+- 0#*.*hhd"; /* Current format spec. */
+ char *p; /* Position within fstart. */
+ unsigned char c; /* A simple character. */
+ int index = 0; /* Index within argc used so far. */
+ bool valid_format = true; /* True if entire format string ok. */
/* Flags. */
- char flags; /* flags to use in fstart */
+ char flags; /* Flags to use in fstart. */
enum {
- THOUSANDS = 0x01, /* ' */
- PLUS = 0x02, /* + */
- MINUS = 0x04, /* - */
- SPACE = 0x08, /* */
- ZERO = 0x10, /* 0 */
- ALT = 0x20, /* # */
- DONE = 0x40 /* no more flags */
+ THOUSANDS = 0x01, /* '\''. */
+ PLUS = 0x02, /* '+'. */
+ MINUS = 0x04, /* '-'. */
+ SPACE = 0x08, /* ' '. */
+ ZERO = 0x10, /* '0'. */
+ ALT = 0x20, /* '#'. */
+ DONE = 0x40 /* No more flags. */
};
/* Precision specifiers. */
- int width; /* minimum field width */
- int prec; /* precision */
- char lflag; /* long flag */
+ int width; /* Minimum field width. */
+ int prec; /* Precision. */
+ char lflag; /* Long flag. */
/* Specifiers we are willing to accept. ok['x'] implies %x is ok.
Various modifiers reduce the set, in order to avoid undefined
@@ -82,17 +156,23 @@ format (struct obstack *obs, int argc, macro_arguments
*argv)
char ok[128];
/* Buffer and stuff. */
- char *str; /* malloc'd buffer of formatted text */
+ char *base; /* Current position in obs. */
+ size_t len; /* Length of formatted text. */
+ char *str; /* Malloc'd buffer of formatted text. */
enum {CHAR, INT, LONG, DOUBLE, STR} datatype;
f = fmt = ARG_STR (index, argc, argv);
memset (ok, 0, sizeof ok);
- for (;;)
+ while (true)
{
while ((c = *fmt++) != '%')
{
if (c == '\0')
- return;
+ {
+ if (valid_format)
+ bad_argc (me, argc, index, index);
+ return;
+ }
obstack_1grow (obs, c);
}
@@ -229,6 +309,7 @@ format (struct obstack *obs, int argc, macro_arguments
*argv)
if (c > sizeof ok || !ok[c])
{
m4_warn (0, me, _("unrecognized specifier in `%s'"), f);
+ valid_format = false;
if (c == '\0')
fmt--;
continue;
@@ -271,42 +352,56 @@ format (struct obstack *obs, int argc, macro_arguments
*argv)
}
*p++ = c;
*p = '\0';
+ base = obstack_next_free (obs);
+ len = obstack_room (obs);
switch (datatype)
{
case CHAR:
- str = xasprintf (fstart, width, ARG_INT (index, argc, argv));
+ str = asnprintf (base, &len, fstart, width,
+ ARG_INT (index, argc, argv));
break;
case INT:
- str = xasprintf (fstart, width, prec, ARG_INT (index, argc, argv));
+ str = asnprintf (base, &len, fstart, width, prec,
+ ARG_INT (index, argc, argv));
break;
case LONG:
- str = xasprintf (fstart, width, prec, ARG_LONG (index, argc, argv));
+ str = asnprintf (base, &len, fstart, width, prec,
+ ARG_LONG (index, argc, argv));
break;
case DOUBLE:
- str = xasprintf (fstart, width, prec, ARG_DOUBLE (index, argc, argv));
+ str = asnprintf (base, &len, fstart, width, prec,
+ ARG_DOUBLE (index, argc, argv));
break;
case STR:
- str = xasprintf (fstart, width, prec, ARG_STR (index, argc, argv));
+ str = asnprintf (base, &len, fstart, width, prec,
+ ARG_STR (index, argc, argv));
break;
default:
abort ();
}
- /* NULL was returned on failure, such as invalid format string.
- Issue a warning, then proceed. */
if (str == NULL)
+ /* NULL is unexpected (EILSEQ and EINVAL are not possible
+ based on our construction of fstart, leaving only ENOMEM,
+ which should always be fatal). */
+ m4_error (EXIT_FAILURE, errno, me,
+ _("unable to format output for `%s'"), f);
+ else if (str == base)
+ /* The output was already computed in place, but we need to
+ account for its size. */
+ obstack_blank_fast (obs, len);
+ else
{
- m4_warn (0, me, _("unable to format output for `%s'"), f);
- continue;
+ /* The output exceeded available obstack space, copy the
+ allocated string. */
+ obstack_grow (obs, str, len);
+ free (str);
}
-
- obstack_grow (obs, str, strlen (str));
- free (str);
}
}
diff --git a/src/input.c b/src/input.c
index 1753be2..bc73c6f 100644
--- a/src/input.c
+++ b/src/input.c
@@ -1,7 +1,7 @@
/* GNU m4 -- A simple macro processor
- Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2004, 2005, 2006, 2007
- Free Software Foundation, Inc.
+ Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2004, 2005, 2006, 2007,
+ 2008 Free Software Foundation, Inc.
This file is part of GNU M4.
@@ -692,17 +692,17 @@ peek_input (void)
}
}
-/*-------------------------------------------------------------------------.
-| The function next_char () is used to read and advance the input to the |
-| next character. It also manages line numbers for error messages, so |
-| they do not get wrong, due to lookahead. The token consisting of a |
-| newline alone is taken as belonging to the line it ends, and the current |
-| line number is not incremented until the next character is read. |
-| 99.9% of all calls will read from a string, so factor that out into a |
-| macro for speed. |
-`-------------------------------------------------------------------------*/
-
-#define next_char() \
+/*-------------------------------------------------------------------.
+| The function next_char () is used to read and advance the input to |
+| the next character. It also manages line numbers for error |
+| messages, so they do not get wrong due to lookahead. The token |
+| consisting of a newline alone is taken as belonging to the line it |
+| ends, and the current line number is not incremented until the |
+| next character is read. 99.9% of all calls will read from a |
+| string, so factor that out into a macro for speed. |
+`-------------------------------------------------------------------*/
+
+#define next_char() \
(isp && isp->type == INPUT_STRING && isp->u.u_s.len && !input_change \
? (isp->u.u_s.len--, to_uchar (*isp->u.u_s.str++)) \
: next_char_1 ())
@@ -883,9 +883,9 @@ match_input (const char *s, bool consume)
| effectively unchanged. |
`--------------------------------------------------------------------*/
-#define MATCH(ch, s, consume) \
- (to_uchar ((s)[0]) == (ch) \
- && (ch) != '\0' \
+#define MATCH(ch, s, consume) \
+ (to_uchar ((s)[0]) == (ch) \
+ && (ch) != '\0' \
&& ((s)[1] == '\0' || (match_input ((s) + (consume), consume))))
@@ -1142,22 +1142,24 @@ safe_quotes (void)
/*--------------------------------------------------------------------.
| Parse and return a single token from the input stream. A token |
| can either be TOKEN_EOF, if the input_stack is empty; it can be |
-| TOKEN_STRING for a quoted string; TOKEN_WORD for something that is |
-| a potential macro name; and TOKEN_SIMPLE for any single character |
-| that is not a part of any of the previous types. If LINE is not |
-| NULL, set *LINE to the line where the token starts. Report errors |
-| (unterminated comments or strings) on behalf of CALLER, if |
-| non-NULL. |
-| |
+| TOKEN_STRING for a quoted string or comment; TOKEN_WORD for |
+| something that is a potential macro name; and TOKEN_SIMPLE for any |
+| single character that is not a part of any of the previous types. |
+| If LINE is not NULL, set *LINE to the line where the token starts. |
+| If OBS is not NULL, expand TOKEN_STRING directly into OBS rather |
+| than in token_stack temporary storage area. Report errors |
+| (unterminated comments or strings) on behalf of CALLER, if |
+| non-NULL. |
+| |
| Next_token () returns the token type, and passes back a pointer to |
-| the token data through TD. The token text is collected on the |
-| obstack token_stack, which never contains more than one token text |
-| at a time. The storage pointed to by the fields in TD is |
+| the token data through TD. Non-string token text is collected on |
+| the obstack token_stack, which never contains more than one token |
+| text at a time. The storage pointed to by the fields in TD is |
| therefore subject to change the next time next_token () is called. |
`--------------------------------------------------------------------*/
token_type
-next_token (token_data *td, int *line, const char *caller)
+next_token (token_data *td, int *line, struct obstack *obs, const char *caller)
{
int ch;
int quote_level;
@@ -1168,6 +1170,11 @@ next_token (token_data *td, int *line, const char
*caller)
#endif /* ENABLE_CHANGEWORD */
const char *file;
int dummy;
+ /* The obstack where token data is stored. Generally token_stack,
+ for tokens where argument collection might not use the literal
+ token. But for comments and strings, we can output directly into
+ the argument collection obstack obs, if one was provided. */
+ struct obstack *obs_td = &token_stack;
obstack_free (&token_stack, token_bottom);
if (!line)
@@ -1199,12 +1206,14 @@ next_token (token_data *td, int *line, const char
*caller)
*line = current_line;
if (MATCH (ch, bcomm.string, true))
{
- obstack_grow (&token_stack, bcomm.string, bcomm.length);
+ if (obs)
+ obs_td = obs;
+ obstack_grow (obs_td, bcomm.string, bcomm.length);
while ((ch = next_char ()) != CHAR_EOF
&& !MATCH (ch, ecomm.string, true))
- obstack_1grow (&token_stack, ch);
+ obstack_1grow (obs_td, ch);
if (ch != CHAR_EOF)
- obstack_grow (&token_stack, ecomm.string, ecomm.length);
+ obstack_grow (obs_td, ecomm.string, ecomm.length);
else
/* Current_file changed to "" if we see CHAR_EOF, use the
previous value we stored earlier. */
@@ -1283,6 +1292,8 @@ next_token (token_data *td, int *line, const char *caller)
}
else
{
+ if (obs)
+ obs_td = obs;
quote_level = 1;
while (1)
{
@@ -1297,23 +1308,28 @@ next_token (token_data *td, int *line, const char
*caller)
{
if (--quote_level == 0)
break;
- obstack_grow (&token_stack, rquote.string, rquote.length);
+ obstack_grow (obs_td, rquote.string, rquote.length);
}
else if (MATCH (ch, lquote.string, true))
{
quote_level++;
- obstack_grow (&token_stack, lquote.string, lquote.length);
+ obstack_grow (obs_td, lquote.string, lquote.length);
}
else
- obstack_1grow (&token_stack, ch);
+ obstack_1grow (obs_td, ch);
}
type = TOKEN_STRING;
}
TOKEN_DATA_TYPE (td) = TOKEN_TEXT;
- TOKEN_DATA_LEN (td) = obstack_object_size (&token_stack);
- obstack_1grow (&token_stack, '\0');
- TOKEN_DATA_TEXT (td) = (char *) obstack_finish (&token_stack);
+ TOKEN_DATA_LEN (td) = obstack_object_size (obs_td);
+ if (obs_td != obs)
+ {
+ obstack_1grow (obs_td, '\0');
+ TOKEN_DATA_TEXT (td) = (char *) obstack_finish (obs_td);
+ }
+ else
+ TOKEN_DATA_TEXT (td) = NULL;
TOKEN_DATA_QUOTE_AGE (td) = current_quote_age;
#ifdef ENABLE_CHANGEWORD
if (orig_text == NULL)
@@ -1455,7 +1471,7 @@ lex_debug (void)
token_type t;
token_data td;
- while ((t = next_token (&td, NULL, "<debug>")) != TOKEN_EOF)
+ while ((t = next_token (&td, NULL, NULL, "<debug>")) != TOKEN_EOF)
print_token ("lex", t, &td);
}
#endif /* DEBUG_INPUT */
diff --git a/src/m4.h b/src/m4.h
index ce78455..ea3947f 100644
--- a/src/m4.h
+++ b/src/m4.h
@@ -1,7 +1,7 @@
/* GNU m4 -- A simple macro processor
- Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2004, 2005, 2006, 2007
- Free Software Foundation, Inc.
+ Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2004, 2005, 2006, 2007,
+ 2008 Free Software Foundation, Inc.
This file is part of GNU M4.
@@ -41,10 +41,12 @@
#include "closein.h"
#include "error.h"
#include "exitfail.h"
+#include "intprops.h"
#include "obstack.h"
#include "stdio--.h"
#include "stdlib--.h"
#include "unistd--.h"
+#include "vasnprintf.h"
#include "verror.h"
#include "xalloc.h"
#include "xprintf.h"
@@ -336,7 +338,7 @@ typedef enum token_data_type token_data_type;
void input_init (void);
token_type peek_token (void);
-token_type next_token (token_data *, int *, const char *);
+token_type next_token (token_data *, int *, struct obstack *, const char *);
void skip_line (const char *);
/* push back input */
@@ -486,6 +488,7 @@ struct re_registers;
#define DEFAULT_MACRO_SEQUENCE "\\$\\({[^}]*}\\|[0-9][0-9]+\\)"
void builtin_init (void);
+bool bad_argc (const char *, int, unsigned int, unsigned int);
void define_builtin (const char *, const builtin *, symbol_lookup);
void set_macro_sequence (const char *);
void free_regex (void);
diff --git a/src/macro.c b/src/macro.c
index 964be5b..ef18b8f 100644
--- a/src/macro.c
+++ b/src/macro.c
@@ -1,6 +1,6 @@
/* GNU m4 -- A simple macro processor
- Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2006, 2007 Free
+ Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2006, 2007, 2008 Free
Software Foundation, Inc.
This file is part of GNU M4.
@@ -212,8 +212,8 @@ expand_input (void)
TOKEN_DATA_ORIG_TEXT (&empty_token) = "";
#endif
- while ((t = next_token (&td, &line, NULL)) != TOKEN_EOF)
- expand_token ((struct obstack *) NULL, t, &td, line, true);
+ while ((t = next_token (&td, &line, NULL, NULL)) != TOKEN_EOF)
+ expand_token (NULL, t, &td, line, true);
for (i = 0; i < stacks_count; i++)
{
@@ -264,8 +264,12 @@ expand_token (struct obstack *obs, token_type t,
token_data *td, int line,
/* Tokens and comments are safe in isolation (since quote_age()
detects any change in delimiters). But if other text is
already present, multi-character delimiters could be an
- issue, so use a conservative heuristic. */
+ issue, so use a conservative heuristic. If obstack is
+ provided, the string was already expanded into it during
+ next_token. */
result = first || safe_quotes ();
+ if (obs)
+ return result;
break;
case TOKEN_OPEN:
@@ -356,7 +360,7 @@ expand_argument (struct obstack *obs, token_data *argp,
const char *caller)
/* Skip leading white space. */
do
{
- t = next_token (&td, NULL, caller);
+ t = next_token (&td, NULL, obs, caller);
}
while (t == TOKEN_SIMPLE && isspace (to_uchar (*TOKEN_DATA_TEXT (&td))));
@@ -432,7 +436,7 @@ expand_argument (struct obstack *obs, token_data *argp,
const char *caller)
if (TOKEN_DATA_TYPE (argp) != TOKEN_VOID || obstack_object_size (obs))
first = false;
- t = next_token (&td, NULL, caller);
+ t = next_token (&td, NULL, obs, caller);
}
}
@@ -464,16 +468,18 @@ collect_arguments (symbol *sym, struct obstack *arguments,
if (peek_token () == TOKEN_OPEN)
{
- next_token (&td, NULL, SYMBOL_NAME (sym)); /* gobble parenthesis */
+ next_token (&td, NULL, NULL, SYMBOL_NAME (sym)); /* gobble parenthesis */
do
{
- more_args = expand_argument (arguments, &td, SYMBOL_NAME (sym));
+ tdp = (token_data *) obstack_alloc (arguments, sizeof *tdp);
+ more_args = expand_argument (arguments, tdp, SYMBOL_NAME (sym));
- if ((TOKEN_DATA_TYPE (&td) == TOKEN_TEXT && !TOKEN_DATA_LEN (&td))
- || (!groks_macro_args && TOKEN_DATA_TYPE (&td) == TOKEN_FUNC))
- tdp = &empty_token;
- else
- tdp = (token_data *) obstack_copy (arguments, &td, sizeof td);
+ if ((TOKEN_DATA_TYPE (tdp) == TOKEN_TEXT && !TOKEN_DATA_LEN (tdp))
+ || (!groks_macro_args && TOKEN_DATA_TYPE (tdp) == TOKEN_FUNC))
+ {
+ obstack_free (arguments, tdp);
+ tdp = &empty_token;
+ }
obstack_ptr_grow (argv_stack, tdp);
args.arraylen++;
args.argc++;
diff --git a/src/output.c b/src/output.c
index 4c8c9de..d252d74 100644
--- a/src/output.c
+++ b/src/output.c
@@ -1,7 +1,7 @@
/* GNU m4 -- A simple macro processor
Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2004, 2005, 2006,
- 2007 Free Software Foundation, Inc.
+ 2007, 2008 Free Software Foundation, Inc.
This file is part of GNU M4.
@@ -188,15 +188,19 @@ static const char *
m4_tmpname (int divnum)
{
static char *buffer;
- static char *tail;
+ static size_t offset;
if (buffer == NULL)
{
- tail = xasprintf ("%s/m4-%d", output_temp_dir->dir_name, INT_MAX);
- buffer = obstack_copy0 (&diversion_storage, tail, strlen (tail));
- free (tail);
- tail = strrchr (buffer, '-') + 1;
+ obstack_grow (&diversion_storage, output_temp_dir->dir_name,
+ strlen (output_temp_dir->dir_name));
+ obstack_1grow (&diversion_storage, '/');
+ obstack_1grow (&diversion_storage, 'm');
+ obstack_1grow (&diversion_storage, '4');
+ obstack_1grow (&diversion_storage, '-');
+ offset = obstack_object_size (&diversion_storage);
+ buffer = obstack_alloc (&diversion_storage, INT_BUFSIZE_BOUND (divnum));
}
- if (sprintf (tail, "%d", divnum) < 0)
+ if (snprintf (&buffer[offset], INT_BUFSIZE_BOUND (divnum), "%d", divnum) < 0)
m4_error (EXIT_FAILURE, errno, NULL,
_("cannot create temporary file for diversion"));
return buffer;
diff --git a/src/symtab.c b/src/symtab.c
index e8a027f..277a79f 100644
--- a/src/symtab.c
+++ b/src/symtab.c
@@ -1,7 +1,7 @@
/* GNU m4 -- A simple macro processor
- Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2003, 2006, 2007 Free
- Software Foundation, Inc.
+ Copyright (C) 1989, 1990, 1991, 1992, 1993, 1994, 2003, 2006, 2007, 2008
+ Free Software Foundation, Inc.
This file is part of GNU M4.
@@ -350,7 +350,7 @@ symtab_debug (void)
int delete;
static int i;
- while (next_token (&td, NULL, "<debug>") == TOKEN_WORD)
+ while (next_token (&td, NULL, NULL, "<debug>") == TOKEN_WORD)
{
text = TOKEN_DATA_TEXT (&td);
if (*text == '_')
--
1.5.3.8
| [Prev in Thread] |
Current Thread |
[Next in Thread] |
- [10/18] argv_ref speedup: avoid extra string copying,
Eric Blake <=