antlr4 gives token recognition error with lexer island grammar

StackOverflow https://stackoverflow.com/questions/21743067

  •  10-10-2022
  •  | 
  •  

質問

I need antlr4 to parse some simple HTML files. I have split my grammar into a parser grammar and a lexer grammar so I could use an island grammar for stuff inside tags (inside < and >) as described in "The Definitive ANTLR4 Reference". antlr4 repeatedly tells me "token recognition error".

parser grammar:

grammar Rule;

options {
    tokenVocab = HTMLLexer;
    language = Java;
}

/* Parser Rules */
doc : type? html ;
type : '<!DOCTYPE HTML>' ;
html : shtml head body ehtml ;

head : shead meta* ehead ;
meta : smeta ;

body : sbody ebody ;

shtml : '<' 'html' attr* '>' ;
ehtml : '<' '/html' '>' ;
shead : '<' 'head' attr* '>' ;
ehead : '<' '/head' '>' ;
smeta : '<' 'meta' attr+ '>' ;

sbody : '<' 'body' attr* '>' ;
ebody : '<' '/body' '>' ;

attr : NAME '=' VALUE ;

lexer grammer:

lexer grammar HTMLLexer;

COMMENT : '<!--' .*? '-->' -> skip ;
CDATA   : '<![CDATA[' .*? ']]>' ;

OPEN      : '<'  -> pushMode(INSIDE) ;
SPEC_OPEN : '<!' -> pushMode(INSIDE) ;

TEXT : (ENTITY | ~[<&])+ ;
fragment ENTITY
    : '&' [a-zA-Z]+ ';'
    | '&#' [0-9]+ ';'
    | '&#x' [0-9A-Za-z]+ ';' ;

mode INSIDE;
CLOSE       : '>'  -> popMode ;
SLASH_CLOSE : '/>' -> popMode ;

StHTML : 'html' ;
EnHTML : '/html' ;

StHead : 'head' ;
EnHead : '/head' ;
StMeta : 'meta' ;

StBody : 'body' ;
EnBody : '/body' ;

NAME : 'class'
    | 'content'
    | 'http-equiv'
    | 'id'
    | 'lang'
    | 'name'
    | 'style'
    | 'type'
    ;

EQUALS : '=' ;

VALUE : ('"' ~["<>\r\n]+ '"')
    | ('\'' ~['<>\r\n]+ '\'')
    | ~["'<>= \t\r\n]+ ;
    ;

WS : [ \t\r\n]+ -> skip ;

sample HTML file:

<html>
<head>
<meta http-equiv=Content-Type content="text/html; charset=windows-1252">
<meta name=Generator content="Microsoft Word 14 (filtered)">
</head>

<body lang=EN-US style='text-justify-trim:punctuation'>
</body>
</html>

output from antlr4:

line 1:6 token recognition error at: '\n'
line 2:6 token recognition error at: '\n'
line 3:5 token recognition error at: ' '
line 3:6 token recognition error at: 'htt'
line 3:9 token recognition error at: 'p'
...
[@0,0:0='<',<7>,1:0]
[@1,1:4='html',<10>,1:1]
[@2,5:5='>',<1>,1:5]
[@3,7:7='<',<7>,2:0]
[@4,8:11='head',<6>,2:1]
[@5,12:12='>',<1>,2:5]
[@6,14:14='<',<7>,3:0]
[@7,15:18='meta',<2>,3:1]
[@8,30:30='=',<9>,3:16]
[@9,51:51='=',<9>,3:37]
[@10,57:61='/html',<4>,3:43]
[@11,71:71='=',<9>,3:57]
[@12,85:85='>',<1>,3:71]
[@13,87:87='<',<7>,4:0]
[@14,88:91='meta',<2>,4:1]
[@15,115:115='=',<9>,4:28]
[@16,146:146='>',<1>,4:59]
[@17,148:148='<',<7>,5:0]
[@18,149:153='/head',<8>,5:1]
[@19,154:154='>',<1>,5:6]
[@20,157:157='<',<7>,7:0]
[@21,158:161='body',<5>,7:1]
[@22,167:167='=',<9>,7:10]
[@23,179:179='=',<9>,7:22]
[@24,211:211='>',<1>,7:54]
[@25,213:213='<',<7>,8:0]
[@26,214:218='/body',<11>,8:1]
[@27,219:219='>',<1>,8:6]
[@28,221:221='<',<7>,9:0]
[@29,222:226='/html',<4>,9:1]
[@30,227:227='>',<1>,9:6]
[@31,229:228='<EOF>',<-1>,10:0]
line 3:16 mismatched input '=' expecting NAME
line 4:28 mismatched input '=' expecting NAME
line 7:10 mismatched input '=' expecting {'>', NAME}
役に立ちましたか？

解決

First of all, you need to change the declaration of your parser to parser grammar Rule; instead of grammar Rule;. I don't see any problems with your lexer that would produce those particular error messages, so that could be the problem.

ライセンス： CC-BY-SA と 帰属
所属していません StackOverflow
scroll top