lex(1p) - phpMan

LEX(1P)                    POSIX Programmer's Manual                   LEX(1P)

PROLOG
       This  manual  page is part of the POSIX Programmer's Manual.  The Linux
       implementation of this interface may differ (consult the  corresponding
       Linux  manual page for details of Linux behavior), or the interface may
       not be implemented on Linux.
NAME
       lex - generate programs for lexical tasks (DEVELOPMENT)
SYNOPSIS
       lex [-t][-n|-v][file ...]
DESCRIPTION
       The lex utility shall generate C programs to be used  in  lexical  pro-
       cessing  of  character  input,  and that can be used as an interface to
       yacc. The C programs shall be generated from lex source code  and  con-
       form  to  the  ISO C standard. Usually, the lex utility shall write the
       program it generates to the file lex.yy.c; the state of  this  file  is
       unspecified  if lex exits with a non-zero exit status. See the EXTENDED
       DESCRIPTION section for a complete description of the  lex  input  lan-
       guage.
OPTIONS
       The  lex  utility  shall  conform  to  the  Base  Definitions volume of
       IEEE Std 1003.1-2001, Section 12.2, Utility Syntax Guidelines.
       The following options shall be supported:
       -n     Suppress the summary of statistics usually written with  the  -v
              option.  If  no table sizes are specified in the lex source code
              and the -v option is not specified, then -n is implied.
       -t     Write the  resulting  program  to  standard  output  instead  of
              lex.yy.c.
       -v     Write  a  summary of lex statistics to the standard output. (See
              the discussion of lex table sizes in Definitions in  lex  .)  If
              the  -t option is specified and -n is not specified, this report
              shall be written to standard error. If table sizes are specified
              in  the  lex source code, and if the -n option is not specified,
              the -v option may be enabled.

OPERANDS
       The following operand shall be supported:
       file   A pathname of an input file. If more than one such file is spec-
              ified,  all  files shall be concatenated to produce a single lex
              program. If no file operands are specified, or if a file operand
              is '-', the standard input shall be used.

STDIN
       The  standard input shall be used if no file operands are specified, or
       if a file operand is '-' . See INPUT FILES.
INPUT FILES
       The input files shall be text files  containing  lex  source  code,  as
       described in the EXTENDED DESCRIPTION section.
ENVIRONMENT VARIABLES
       The following environment variables shall affect the execution of lex:
       LANG   Provide  a  default value for the internationalization variables
              that are unset or null. (See  the  Base  Definitions  volume  of
              IEEE Std 1003.1-2001,  Section  8.2,  Internationalization Vari-
              ables for the precedence of internationalization variables  used
              to determine the values of locale categories.)
       LC_ALL If  set  to a non-empty string value, override the values of all
              the other internationalization variables.
       LC_COLLATE
              Determine the locale for the  behavior  of  ranges,  equivalence
              classes,  and  multi-character collating elements within regular
              expressions. If this variable is not set to  the  POSIX  locale,
              the results are unspecified.
       LC_CTYPE
              Determine  the  locale  for  the  interpretation of sequences of
              bytes of text data as characters (for  example,  single-byte  as
              opposed  to multi-byte characters in arguments and input files),
              and the behavior of character  classes  within  regular  expres-
              sions.   If  this  variable  is not set to the POSIX locale, the
              results are unspecified.
       LC_MESSAGES
              Determine the locale that should be used to  affect  the  format
              and contents of diagnostic messages written to standard error.
       NLSPATH
              Determine the location of message catalogs for the processing of
              LC_MESSAGES .

ASYNCHRONOUS EVENTS
       Default.
STDOUT
       If the -t option is specified, the text file of C source code output of
       lex shall be written to standard output.
       If the -t option is not specified:
        * Implementation-defined  informational,  error,  and warning messages
          concerning the contents of lex source code input shall be written to
          either the standard output or standard error.
        * If  the  -v  option is specified and the -n option is not specified,
          lex statistics shall also be written to either the  standard  output
          or  standard  error, in an implementation-defined format. These sta-
          tistics may also be generated if table sizes are  specified  with  a
          '%' operator in the Definitions section, as long as the -n option is
          not specified.
STDERR
       If the -t option is  specified,  implementation-defined  informational,
       error,  and warning messages concerning the contents of lex source code
       input shall be written to the standard error.
       If the -t option is not specified:
        1. Implementation-defined informational, error, and  warning  messages
           concerning  the  contents of lex source code input shall be written
           to either the standard output or standard error.
        2. If the -v option is specified and the -n option is  not  specified,
           lex  statistics shall also be written to either the standard output
           or standard error, in an implementation-defined format. These  sta-
           tistics  may  also be generated if table sizes are specified with a
           '%' operator in the Definitions section, as long as the  -n  option
           is not specified.
OUTPUT FILES
       A  text  file containing C source code shall be written to lex.yy.c, or
       to the standard output if the -t option is present.
EXTENDED DESCRIPTION
       Each input file shall contain lex source code, which is a table of reg-
       ular  expressions  with  corresponding actions in the form of C program
       fragments.
       When lex.yy.c is compiled and linked with the lex  library  (using  the
       -l l  operand  with  c99),  the  resulting program shall read character
       input from the standard input and shall partition it into strings  that
       match the given expressions.
       When an expression is matched, these actions shall occur:
        * The input string that was matched shall be left in yytext as a null-
          terminated string; yytext shall  either  be  an  external  character
          array  or  a  pointer to a character string. As explained in Defini-
          tions in lex, the type can be explicitly selected using  the  %array
          or %pointer declarations, but the default is implementation-defined.
        * The  external  int yyleng shall be set to the length of the matching
          string.
        * The expression's corresponding program fragment, or action, shall be
          executed.
       During  pattern  matching, lex shall search the set of patterns for the
       single longest possible match. Among rules that match the  same  number
       of characters, the rule given first shall be chosen.
       The general format of lex source shall be:

              Definitions
              %%
              Rules
              %%
              UserSubroutines
       The  first "%%" is required to mark the beginning of the rules (regular
       expressions and actions); the second "%%" is required only if user sub-
       routines follow.
       Any  line  in the Definitions section beginning with a <blank> shall be
       assumed to be a C program fragment and shall be copied to the  external
       definition area of the lex.yy.c file.  Similarly, anything in the Defi-
       nitions section included between delimiter lines containing  only  "%{"
       and "%}" shall also be copied unchanged to the external definition area
       of the lex.yy.c file.
       Any such input (beginning with a <blank> or within "%{" and "%}" delim-
       iter  lines) appearing at the beginning of the Rules section before any
       rules are specified shall be written to lex.yy.c after the declarations
       of variables for the yylex() function and before the first line of code
       in yylex(). Thus, user variables local to yylex() can be declared here,
       as well as application code to execute upon entry to yylex().
       The  action  taken  by lex when encountering any input beginning with a
       <blank> or within "%{" and "%}" delimiter lines appearing in the  Rules
       section  but  coming after one or more rules is undefined. The presence
       of such input may result in an  erroneous  definition  of  the  yylex()
       function.
   Definitions in lex
       Definitions  appear  before  the first "%%" delimiter. Any line in this
       section not contained between "%{" and "%}"  lines  and  not  beginning
       with  a  <blank>  shall be assumed to define a lex substitution string.
       The format of these lines shall be:

              name substitute
       If a name does not meet the requirements for identifiers in  the  ISO C
       standard,  the result is undefined. The string substitute shall replace
       the string { name} when it is used in a rule. The name string shall  be
       recognized  in  this context only when the braces are provided and when
       it does not appear within a bracket expression or within double-quotes.
       In the Definitions section, any line  beginning  with  a  '%'  (percent
       sign)  character  and  followed  by an alphanumeric word beginning with
       either 's' or 'S' shall define a set  of  start  conditions.  Any  line
       beginning  with  a  '%' followed by a word beginning with either 'x' or
       'X' shall define a set of exclusive start conditions. When  the  gener-
       ated  scanner  is in a %s state, patterns with no state specified shall
       be also active; in a %x state, such patterns shall not be  active.  The
       rest  of  the line, after the first word, shall be considered to be one
       or more <blank>-separated names of start  conditions.  Start  condition
       names  shall  be constructed in the same way as definition names. Start
       conditions can be used to restrict the matching of regular  expressions
       to one or more states as described in Regular Expressions in lex .
       Implementations  shall  accept  either  of  the following two mutually-
       exclusive declarations in the Definitions section:
       %array Declare the type of yytext to  be  a  null-terminated  character
              array.
       %pointer
              Declare  the type of yytext to be a pointer to a null-terminated
              character string.

       The default type of yytext is implementation-defined. If an application
       refers  to  yytext  outside of the scanner source file (that is, via an
       extern), the  application  shall  include  the  appropriate  %array  or
       %pointer declaration in the scanner source file.
       Implementations  shall  accept  declarations in the Definitions section
       for setting certain internal table sizes. The declarations are shown in
       the following table.
                        Table: Table Size Declarations in lex
           Declaration  Description                         Minimum Value
           %p n         Number of positions                 2500
           %n n         Number of states                    500
           %a n         Number of transitions               2000
           %e n         Number of parse tree nodes          1000
           %k n         Number of packed character classes  1000
           %o n         Size of the output array            3000
       In  the table, n represents a positive decimal integer, preceded by one
       or more <blank>s. The exact meaning of  these  table  size  numbers  is
       implementation-defined.  The  implementation  shall  document how these
       numbers affect the lex utility and how they are related to  any  output
       that  may  be  generated  by  the  implementation should limitations be
       encountered during the execution of lex. It shall be possible to deter-
       mine  from this output which of the table size values needs to be modi-
       fied to permit lex to successfully generate tables for the  input  lan-
       guage.   The  values  in  the column Minimum Value represent the lowest
       values conforming implementations shall provide.
   Rules in lex
       The rules in lex source files are a table in which the left column con-
       tains regular expressions and the right column contains actions (C pro-
       gram fragments) to be executed when the expressions are recognized.

              ERE action
              ERE action...
       The extended regular expression (ERE) portion of a row shall  be  sepa-
       rated  from  action  by one or more <blank>s. A regular expression con-
       taining <blank>s shall be recognized under one of the following  condi-
       tions:
        * The entire expression appears within double-quotes.
        * The <blank>s appear within double-quotes or square brackets.
        * Each <blank> is preceded by a backslash character.
   User Subroutines in lex
       Anything  in  the  user subroutines section shall be copied to lex.yy.c
       following yylex().
   Regular Expressions in lex
       The lex utility shall support the set of extended  regular  expressions
       (see  the Base Definitions volume of IEEE Std 1003.1-2001, Section 9.4,
       Extended Regular Expressions), with the following additions and  excep-
       tions to the syntax:
       "..."  Any string enclosed in double-quotes shall represent the charac-
              ters within the double-quotes as themselves, except  that  back-
              slash  escapes  (which  appear  in the following table) shall be
              recognized.  Any backslash-escape sequence shall  be  terminated
              by the closing quote. For example, "\01" "1" represents a single
              string: the octal value 1 followed by the character '1' .
       <state>r, <state1,state2,...>r
              The regular expression r shall be matched only when the  program
              is  in  one  of the start conditions indicated by state, state1,
              and so on; see Actions in lex . (As an exception  to  the  typo-
              graphical   conventions   of   the   rest   of  this  volume  of
              IEEE Std 1003.1-2001, in this case <state> does not represent  a
              metavariable, but the literal angle-bracket characters surround-
              ing a symbol.) The start condition shall be recognized  as  such
              only at the beginning of a regular expression.
       r/x    The regular expression r shall be matched only if it is followed
              by an occurrence of regular expression x ( x is the instance  of
              trailing context, further defined below).  The token returned in
              yytext shall only match r. If the trailing portion of r  matches
              the  beginning of x, the result is unspecified. The r expression
              cannot include further trailing context or the  '$'  (match-end-
              of-line) operator; x cannot include the '^' (match-beginning-of-
              line) operator, nor trailing context, nor the '$' operator. That
              is,  only one occurrence of trailing context is allowed in a lex
              regular expression, and the '^' operator only can be used at the
              beginning of such an expression.
       {name} When  name  is  one of the substitution symbols from the Defini-
              tions section, the string, including the enclosing braces, shall
              be  replaced by the substitute value. The substitute value shall
              be treated in the extended regular  expression  as  if  it  were
              enclosed  in parentheses. No substitution shall occur if { name}
              occurs within a bracket expression or within double-quotes.

       Within an ERE, a backslash character shall be considered  to  begin  an
       escape  sequence as specified in the table in the Base Definitions vol-
       ume of IEEE Std 1003.1-2001, Chapter 5, File Format  Notation  (  '\\',
       '\a',  '\b',  '\f',  '\n',  '\r', '\t', '\v' ). In addition, the escape
       sequences in the following table shall be recognized.
       A literal <newline> cannot occur within an  ERE;  the  escape  sequence
       '\n'  can  be  used  to represent a <newline>. A <newline> shall not be
       matched by a period operator.
                           Table: Escape Sequences in lex
       Escape
       Sequence Description                    Meaning
       \digits  A backslash character followed The character whose encoding
                by the longest sequence of     is represented by the one,
                one, two, or three octal-digit two, or three-digit octal
                characters (01234567). If all  integer. If the size of a byte
                of the digits are 0 (that is,  on the system is greater than
                representation of the NUL      nine bits, the valid escape
                character), the behavior is    sequence used to represent a
                undefined.                     byte is implementation-
                                               defined. Multi-byte characters
                                               require multiple, concatenated
                                               escape sequences of this type,
                                               including the leading '\' for
                                               each byte.
       \xdigits A backslash character followed The character whose encoding
                by the longest sequence of     is represented by the hexadec-
                hexadecimal-digit characters   imal integer.
                (01234567abcdefABCDEF). If all
                of the digits are 0 (that is,
                representation of the NUL
                character), the behavior is
                undefined.
       \c       A backslash character followed The character 'c', unchanged.
                by any character not described
                in this table or in the table
                in the Base Definitions volume
                of IEEE Std 1003.1-2001, Chap-
                ter 5, File Format Notation (
                '\\', '\a', '\b', '\f', '\n',
                '\r', '\t', '\v' ).
       Note:  If  a  '\x' sequence needs to be immediately followed by a hexa-
              decimal digit character, a sequence such as  "\x1"  "1"  can  be
              used,  which represents a character containing the value 1, fol-
              lowed by the character '1' .

       The order of precedence given to extended regular expressions  for  lex
       differs   from  that  specified  in  the  Base  Definitions  volume  of
       IEEE Std 1003.1-2001, Section 9.4, Extended  Regular  Expressions.  The
       order  of  precedence for lex shall be as shown in the following table,
       from high to low.
       Note:  The escaped characters entry is not meant to  imply  that  these
              are  operators, but they are included in the table to show their
              relationships to the true operators. The start condition, trail-
              ing  context, and anchoring notations have been omitted from the
              table because of the placement restrictions  described  in  this
              section;  they  can only appear at the beginning or ending of an
              ERE.

                                Table: ERE Precedence in lex
                  Extended Regular Expression        Precedence
                  collation-related bracket symbols  [= =] [: :] [. .]
                  escaped characters                 \<special character>
                  bracket expression                 [ ]
                  quoting                            "..."
                  grouping                           ( )
                  definition                         {name}
                  single-character RE duplication    * + ?
                  concatenation
                  interval expression                {m,n}
                  alternation                        |
       The ERE anchoring operators '^' and '$' do not  appear  in  the  table.
       With  lex  regular expressions, these operators are restricted in their
       use: the '^' operator can only be used at the beginning  of  an  entire
       regular expression, and the '$' operator only at the end. The operators
       apply to the entire regular expression. Thus, for example, the  pattern
       "(^abc)|(def$)" is undefined; it can instead be written as two separate
       rules, one with the regular expression  "^abc"  and  one  with  "def$",
       which  share a common action via the special '|' action (see below). If
       the pattern were written "^abc|def$", it would match  either  "abc"  or
       "def" on a line by itself.
       Unlike the general ERE rules, embedded anchoring is not allowed by most
       historical lex implementations. An example of embedded anchoring  would
       be  for  patterns such as "(^| )foo( |$)" to match "foo" when it exists
       as a complete word. This functionality can be obtained  using  existing
       lex features:

              ^foo/[ \n]      |
              " foo"/[ \n]    /* Found foo as a separate word. */
       Note  also  that '$' is a form of trailing context (it is equivalent to
       "/\n" ) and as such cannot be used with regular expressions  containing
       another  instance  of  the  operator  (see  the preceding discussion of
       trailing context).
       The additional regular expressions trailing-context operator '/' can be
       used  as an ordinary character if presented within double-quotes, "/" ;
       preceded by a backslash, "\/" ; or within a bracket expression, "[/]" .
       The  start-condition  '<'  and '>' operators shall be special only in a
       start condition at the beginning of a regular expression; elsewhere  in
       the regular expression they shall be treated as ordinary characters.
   Actions in lex
       The  action to be taken when an ERE is matched can be a C program frag-
       ment or the special actions described below; the program  fragment  can
       contain one or more C statements, and can also include special actions.
       The empty C statement ';' shall be a valid action; any  string  in  the
       lex.yy.c  input  that  matches  the  pattern  portion of such a rule is
       effectively ignored or skipped. However, the absence of an action shall
       not  be  valid,  and  the action lex takes in such a condition is unde-
       fined.
       The specification for an action, including  C  statements  and  special
       actions, can extend across several lines if enclosed in braces:

              ERE <one or more blanks> { program statement
                                         program statement }
       The  default action when a string in the input to a lex.yy.c program is
       not matched by any expression shall be to copy the string to  the  out-
       put.  Because  the default behavior of a program generated by lex is to
       read the input and copy it to the output, a minimal lex source  program
       that  has  just  "%%" shall generate a C program that simply copies the
       input to the output unchanged.
       Four special actions shall be available:

              |   ECHO;   REJECT;   BEGIN
       |      The action '|' means that the action for the next  rule  is  the
              action for this rule. Unlike the other three actions, '|' cannot
              be enclosed in braces or be semicolon-terminated;  the  applica-
              tion  shall  ensure  that  it  is specified alone, with no other
              actions.
       ECHO;  Write the contents of the string yytext on the output.
       REJECT;
              Usually only a single expression is matched by a given string in
              the  input.  REJECT  means "continue to the next expression that
              matches the current input", and shall cause  whatever  rule  was
              the  second choice after the current rule to be executed for the
              same input. Thus, multiple rules can be matched and executed for
              one  input  string  or  overlapping input strings.  For example,
              given the regular expressions  "xyz"  and  "xy"  and  the  input
              "xyz",  usually  only  the regular expression "xyz" would match.
              The next attempted match would start after z. If the last action
              in  the  "xyz"  rule is REJECT, both this rule and the "xy" rule
              would be executed. The REJECT action may be implemented in  such
              a fashion that flow of control does not continue after it, as if
              it were equivalent to a goto to another part of yylex(). The use
              of REJECT may result in somewhat larger and slower scanners.
       BEGIN  The action:

              BEGIN newstate;
       switches  the  state  (start condition) to newstate. If the string new-
       state has not been declared previously as a start condition in the Def-
       initions  section,  the  results  are unspecified. The initial state is
       indicated by the digit '0' or the token INITIAL.

       The functions or macros described below are  accessible  to  user  code
       included in the lex input. It is unspecified whether they appear in the
       C code output of lex, or are accessible only through the  -l l  operand
       to c99 (the lex library).
       int  yylex(void)
              Performs  lexical  analysis  on  the  input; this is the primary
              function generated by the lex utility. The function shall return
              zero  when  the  end  of  input  is reached; otherwise, it shall
              return non-zero values (tokens) determined by the  actions  that
              are selected.
       int  yymore(void)
              When called, indicates that when the next input string is recog-
              nized, it is to be appended  to  the  current  value  of  yytext
              rather  than replacing it; the value in yyleng shall be adjusted
              accordingly.
       int  yyless(int  n)
              Retains n initial  characters  in  yytext,  NUL-terminated,  and
              treats  the  remaining  characters as if they had not been read;
              the value in yyleng shall be adjusted accordingly.
       int  input(void)
              Returns the next character from the input, or  zero  on  end-of-
              file.   It  shall  obtain  input  from  the stream pointer yyin,
              although possibly via an intermediate buffer. Thus,  once  scan-
              ning  has  begun,  the  effect  of altering the value of yyin is
              undefined. The character read shall be removed  from  the  input
              stream of the scanner without any processing by the scanner.
       int  unput(int  c)
              Returns  the  character  'c' to the input; yytext and yyleng are
              undefined until the next expression is matched.  The  result  of
              using  unput()  for  more  characters  than  have  been input is
              unspecified.

       The following functions shall appear only in the lex library accessible
       through the -l l operand; they can therefore be redefined by a conform-
       ing application:
       int  yywrap(void)
              Called by yylex() at end-of-file;  the  default  yywrap()  shall
              always return 1. If the application requires yylex() to continue
              processing with another source of input,  then  the  application
              can  include  a function yywrap(), which associates another file
              with the external variable FILE * yyin and shall return a  value
              of zero.
       int  main(int  argc, char *argv[])
              Calls  yylex() to perform lexical analysis, then exits. The user
              code can contain main() to perform  application-specific  opera-
              tions, calling yylex() as applicable.

       Except  for input(), unput(), and main(), all external and static names
       generated by lex shall begin with the prefix yy or YY.
EXIT STATUS
       The following exit values shall be returned:
        0     Successful completion.
       >0     An error occurred.

CONSEQUENCES OF ERRORS
       Default.
       The following sections are informative.
APPLICATION USAGE
       Conforming applications are warned that in the Rules  section,  an  ERE
       without  an action is not acceptable, but need not be detected as erro-
       neous by lex. This may result in compilation or runtime errors.
       The purpose of input() is to take characters off the input  stream  and
       discard  them as far as the lexical analysis is concerned. A common use
       is to discard the body of a comment once the beginning of a comment  is
       recognized.
       The lex utility is not fully internationalized in its treatment of reg-
       ular expressions in the lex source code or generated lexical  analyzer.
       It would seem desirable to have the lexical analyzer interpret the reg-
       ular expressions given in the lex source according to  the  environment
       specified when the lexical analyzer is executed, but this is not possi-
       ble with the current lex technology. Furthermore, the  very  nature  of
       the lexical analyzers produced by lex must be closely tied to the lexi-
       cal requirements of the input language being described, which  is  fre-
       quently  locale-specific anyway. (For example, writing an analyzer that
       is used for French text is  not  automatically  useful  for  processing
       other languages.)
EXAMPLES
       The following is an example of a lex program that implements a rudimen-
       tary scanner for a Pascal-like syntax:

              %{
              /* Need this for the call to atof() below. */
              #include <math.h>
              /* Need this for printf(), fopen(), and stdin below. */
              #include <stdio.h>
              %}

              DIGIT    [0-9]
              ID       [a-z][a-z0-9]*

              %%

              {DIGIT}+ {
                  printf("An integer: %s (%d)\n", yytext,
                      atoi(yytext));
                  }

              {DIGIT}+"."{DIGIT}*        {
                  printf("A float: %s (%g)\n", yytext,
                      atof(yytext));
                  }

              if|then|begin|end|procedure|function        {
                  printf("A keyword: %s\n", yytext);
                  }

              {ID}    printf("An identifier: %s\n", yytext);

              "+"|"-"|"*"|"/"        printf("An operator: %s\n", yytext);

              "{"[^}\n]*"}"    /* Eat up one-line comments. */

              [ \t\n]+        /* Eat up white space. */

              .  printf("Unrecognized character: %s\n", yytext);

              %%

              int main(int argc, char *argv[])
              {
                  ++argv, --argc;  /* Skip over program name. */
                  if (argc > 0)
                      yyin = fopen(argv[0], "r");
                  else
                      yyin = stdin;

                  yylex();
              }
RATIONALE
       Even though the -c option and references to the C language are retained
       in  this description, lex may be generalized to other languages, as was
       done at one time for EFL, the Extended FORTRAN Language. Since the  lex
       input  specification  is  essentially language-independent, versions of
       this utility could be written to produce Ada, Modula-2, or Pascal code,
       and there are known historical implementations that do so.
       The  current  description  of  lex  bypasses  the issue of dealing with
       internationalized EREs in the lex source code or generated lexical ana-
       lyzer.  If it follows the model used by awk (the source code is assumed
       to be presented in the POSIX locale, but input and output  are  in  the
       locale  specified by the environment variables), then the tables in the
       lexical analyzer produced by lex would interpret EREs specified in  the
       lex source in terms of the environment variables specified when lex was
       executed. The desired effect would be  to  have  the  lexical  analyzer
       interpret the EREs given in the lex source according to the environment
       specified when the lexical analyzer is executed, but this is not possi-
       ble with the current lex technology.
       The  description of octal and hexadecimal-digit escape sequences agrees
       with the ISO C standard use of escape sequences. See the RATIONALE  for
       ed  for  a  discussion of bytes larger than 9 bits being represented by
       octal values.  Hexadecimal values can represent larger bytes and multi-
       byte characters directly, using as many digits as required.
       There is no detailed output format specification. The observed behavior
       of lex under four different historical implementations was that none of
       these  implementations consistently reported the line numbers for error
       and warning messages.  Furthermore, there was  a  desire  that  lex  be
       allowed  to output additional diagnostic messages. Leaving message for-
       mats unspecified avoids these formatting questions  and  problems  with
       internationalization.
       Although the %x specifier for exclusive start conditions is not histor-
       ical practice, it is believed to be a minor change to historical imple-
       mentations  and greatly enhances the usability of lex programs since it
       permits an application to obtain the expected functionality with  fewer
       statements.
       The %array and %pointer declarations were added as a compromise between
       historical systems. The System V-based lex copies the matched text to a
       yytext  array. The flex program, supported in BSD and GNU systems, uses
       a pointer. In the latter case, significant performance improvements are
       available for some scanners. Most historical programs should require no
       change in porting from one system to another because the  string  being
       referenced  is  null-terminated in both cases. (The method used by flex
       in its case is to null-terminate the token in place by remembering  the
       character  that  used  to  come  right after the token and replacing it
       before continuing on to the next scan.) Multi-file programs with exter-
       nal  references  to  yytext outside the scanner source file should con-
       tinue to operate on their historical systems, but would require one  of
       the new declarations to be considered strictly portable.
       The  description  of EREs avoids unnecessary duplication of ERE details
       because their meanings within a lex ERE are the same as  that  for  the
       ERE in this volume of IEEE Std 1003.1-2001.
       The  reason  for the undefined condition associated with text beginning
       with a <blank> or within "%{" and "%}" delimiter lines appearing in the
       Rules  section  is  historical  practice. Both the BSD and System V lex
       copy the indented (or enclosed) input in the Rules section  (except  at
       the  beginning)  to unreachable areas of the yylex() function (the code
       is written directly after a break statement). In some cases, the System
       V  lex  generates  an error message or a syntax error, depending on the
       form of indented input.
       The intention in breaking the list of functions  into  those  that  may
       appear in lex.yy.c versus those that only appear in libl.a is that only
       those functions in libl.a can be reliably  redefined  by  a  conforming
       application.
       The  descriptions  of  standard  output and standard error are somewhat
       complicated because historical lex implementations chose to issue diag-
       nostic   messages   to   standard   output   (unless   -t  was  given).
       IEEE Std 1003.1-2001 allows this behavior, but leaves  an  opening  for
       the  more  expected  behavior  of using standard error for diagnostics.
       Also, the System V behavior of writing the statistics  when  any  table
       sizes are given is allowed, while BSD-derived systems can avoid it. The
       programmer can always precisely obtain the  desired  results  by  using
       either the -t or -n options.
       The  OPERANDS  section  does  not mention the use of - as a synonym for
       standard input; not all historical implementations support  such  usage
       for any of the file operands.
       A description of the translation table was deleted from early proposals
       because of its relatively low usage in historical applications.
       The change to the  definition  of  the  input()  function  that  allows
       buffering of input presents the opportunity for major performance gains
       in some applications.
       The following examples clarify  the  differences  between  lex  regular
       expressions  and regular expressions appearing elsewhere in this volume
       of IEEE Std 1003.1-2001. For regular expressions of the form "r/x", the
       string  matching  r  is  always  returned; confusion may arise when the
       beginning of x matches the trailing portion of r.  For  example,  given
       the  regular  expression  "a*b/cc" and the input "aaabcc", yytext would
       contain the string "aaab" on this match. But given the regular  expres-
       sion  "x*/xy"  and the input "xxxy", the token xxx, not xx, is returned
       by some implementations because xxx matches "x*" .
       In the rule "ab*/bc", the "b*" at the end of r extends r's  match  into
       the beginning of the trailing context, so the result is unspecified. If
       this rule were "ab/bc", however, the rule matches the text "ab" when it
       is  followed  by the text "bc" . In this latter case, the matching of r
       cannot extend into the beginning of x, so the result is specified.
FUTURE DIRECTIONS
       None.
SEE ALSO
       c99, ed, yacc
COPYRIGHT
       Portions of this text are reprinted and reproduced in  electronic  form
       from IEEE Std 1003.1, 2003 Edition, Standard for Information Technology
       -- Portable Operating System Interface (POSIX),  The  Open  Group  Base
       Specifications  Issue  6,  Copyright  (C) 2001-2003 by the Institute of
       Electrical and Electronics Engineers, Inc and The Open  Group.  In  the
       event of any discrepancy between this version and the original IEEE and
       The Open Group Standard, the original IEEE and The Open Group  Standard
       is  the  referee document. The original Standard can be obtained online
       at http://www.opengroup.org/unix/online.html .

IEEE/The Open Group                  2003                              LEX(1P)