,2:0]
Using the option -gui we can also have a nice, and easier to
understand, graphical representation.
ANTLR Example Parse Tree
The dot in mid air represents whitespace.
This works, but it is not very smart or nice, or organized. But don't
worry, we are going to see a better way later. One positive aspect of
this solution is that it allows to show another trick.
TEXT : ('['|'(') .*? (']'|')');
This is an equivalent formulation of the token TEXT: the . matches
any character, * says that the preceding match can be repeated any
time, ? indicates that the previous match is non-greedy. That is to
say the previous sub-rule matches everything except what follows it,
allowing to match the closing parenthesis or square bracket.
Mid-Level
Strumenta Logo
Does it looks like too much work?
...Here there are our ANTLR Services
We can design parsers for new languages, or rewrite parsers for
existing languages built in house.
And much more...
ANTLR Services
In this section we see how to use ANTLR in your programs, the
libraries and functions you need to use, how to test your parsers,
and the like. We see what is and how to use a listener. We also build
up on our knowledge of the basics, by looking at more advanced
concepts, such as semantic predicates. While our projects are mainly
in JavaScript and Python, the concept are generally applicable to
every language. You can come back to this section when you need to
remember how to get your project organized.
12. Setting Up the Chat Project with JavaScript
In the previous sections we have seen how to build a grammar for a
chat program , piece by piece. Let's now copy that grammar we just
created in the same folder of our Javascript files.
grammar Chat;
/*
* Parser Rules
*/
chat : line+ EOF ;
line : name command message NEWLINE ;
message : (emoticon | link | color | mention | WORD | WHITESPACE)+ ;
name : WORD WHITESPACE;
command : (SAYS | SHOUTS) ':' WHITESPACE ;
emoticon : ':' '-'? ')'
| ':' '-'? '('
;
link : TEXT TEXT ;
color : '/' WORD '/' message '/';
mention : '@' WORD ;
/*
* Lexer Rules
*/
fragment A : ('A'|'a') ;
fragment S : ('S'|'s') ;
fragment Y : ('Y'|'y') ;
fragment H : ('H'|'h') ;
fragment O : ('O'|'o') ;
fragment U : ('U'|'u') ;
fragment T : ('T'|'t') ;
fragment LOWERCASE : [a-z] ;
fragment UPPERCASE : [A-Z] ;
SAYS : S A Y S ;
SHOUTS : S H O U T S ;
WORD : (LOWERCASE | UPPERCASE | '_')+ ;
WHITESPACE : (' ' | '\t')+ ;
NEWLINE : ('\r'? '\n' | '\r')+ ;
TEXT : ('['|'(') ~[\])]+ (']'|')');
We can create the corresponding Javascript parser simply by
specifying the correct option with the ANTLR4 Java program.
antlr4 -Dlanguage=JavaScript Chat.g4
Now you will find some new files in the folder, with names such as
ChatLexer.js, ChatParser.js and there are also *.tokens files, none
of which contains anything interesting for us, unless you want to
understand the inner workings of ANTLR.
The file you want to look at is ChatListener.js, you are not going
to modify anything in it, but it contains methods and functions that
we will override with our own listener. We are not going to modify it
because changes would be overwritten every time the grammar is
regenerated.
Looking into it, you can see several enter/exit functions, a pair for
each of our parser rules. These functions will be invoked when a
piece of code matching the rule is encountered. This is the default
implementation of the listener that allows you to just override the
functions that you need, on your derived listener, and leave the rest
as is.
import antlr4 from 'antlr4';
// This class defines a complete listener for a parse tree produced by ChatParser.
export default class ChatListener extends antlr4.tree.ParseTreeListener {
// Enter a parse tree produced by ChatParser#chat.
enterChat(ctx) {
}
// Exit a parse tree produced by ChatParser#chat.
exitChat(ctx) {
}
[..]
The alternative to creating a Listener is creating a Visitor. The
main differences are that you can neither control the flow of a
listener nor return anything from its functions, while you can do
both of them with a visitor. So if you need to control how the nodes
of the parse tree are entered, or to gather information from several
of them, you probably want to use a visitor. This is useful, for
example, with code generation, where some information that is needed
to create new source code is spread around many parts. Both the
listener and the visitor use depth-first search.
A depth-first search means that when a node is accessed, its children
will be accessed, and if one of the children nodes has its own
children, they will be accessed before continuing on with the other
children of the first node. The following image will make the concept
simpler to understand.
Depth-first search
So in the case of a listener, an enter event will be fired at the
first encounter with the node and an exit one will be fired after
having exited all of its children. In the following image you can see
the example of what functions will be fired when a listener would met
a line node (for simplicity only the functions related to line are
shown).
ANTLR Listener Example
With a standard visitor, the behavior will be analogous except, of
course, that only a single visit event will be fired for every single
node. In the following image you can see the example of what function
will be fired when a visitor meets a line node (for simplicity only
the function related to line is shown).
ANTLR Visitor Example
Remember that this is true for the default implementation of a
visitor and it's done by returning the children of each node in every
function. If you override a method of the visitor, it's your
responsibility to make it continuing the journey or stop it right
there.
13. Antlr.js
It is finally time to see how a typical ANTLR program looks.
import { createServer } from 'http';
import antlr4 from 'antlr4';
const { CommonTokenStream, InputStream } = antlr4;
import ChatLexer from './ChatLexer.js';
import ChatParser from './ChatParser.js';
import HtmlChatListener from './HtmlChatListener.js';
createServer((req, res) => {
res.writeHead(200, {
'Content-Type': 'text/html',
});
res.write('');
var input = "john SHOUTS: hello @michael /pink/this will work/ :-) \n";
var chars = new InputStream(input, true)
var lexer = new ChatLexer(chars);
var tokens = new CommonTokenStream(lexer);
var parser = new ChatParser(tokens);
parser.buildParseTrees = true;
var tree = parser.chat();
var htmlChat = new HtmlChatListener(res);
antlr4.tree.ParseTreeWalker.DEFAULT.walk(htmlChat, tree);
res.write('');
res.end();
}).listen(1337);
At the beginning of the main file we import the necessary libraries
and file, antlr4 (the runtime) and our generated parser, plus the
listener that we are going to see later.
For simplicity, we get the input from a string, while in a real
scenario it would come from an editor.
Lines 17-20 shows the foundation of every ANTLR program: you
create the stream of chars from the input, you give it to the
lexer and it transforms them in tokens, that are then interpreted
by the parser.
It is useful to take a moment to reflect on this: the lexer works on
the characters of the input, a copy of the input to be precise, while
the parser works on the tokens generated by the lexer. The lexer does
not work on the input directly, and the parser does not even see the
characters.
This is important to remember in case you need to do something
advanced like manipulating the input. In this case the input is a
string, but, of course, it could be any stream of content.
The line 22 is redundant, since the option already defaults to true,
but it shows that you can enable or disable it.
Then, on line 23, we set the root node of the tree as a chat rule.
You want to invoke the parser specifying a rule, which typically is
the first rule. However, you can actually invoke any rule directly,
like color.
Once we get the parse tree from the parser, typically we want to
process it using a listener or a visitor. In this case we specify a
listener. Our particular listener takes a parameter: the response
object. We want to use it to put some text in the response to send to
the user. After setting the listener up, we finally walk the tree
with our listener.
14. HtmlChatListener.js
We continue by looking at the listener for our Chat project.
import antlr4 from 'antlr4';
import ChatLexer from './ChatLexer.js';
import ChatParser from './ChatParser.js';
import ChatListener from './ChatListener.js';
export default class HtmlChatListener extends ChatListener {
constructor(res) {
super();
this.Res = res;
}
enterName(ctx) {
this.Res.write("");
}
exitName(ctx) {
this.Res.write(ctx.WORD().getText());
this.Res.write(" ");
}
exitEmoticon(ctx) {
var emoticon = ctx.getText();
if(emoticon == ':-)' || emoticon == ':)')
{
ctx.text = "";
}
if(emoticon == ':-(' || emoticon == ':(')
{
ctx.text = "";
}
this.Res.write(ctx.text);
}
enterCommand(ctx) {
if(ctx.SAYS() != null)
this.Res.write(ctx.SAYS().getText() + ':' + '');
if(ctx.SHOUTS() != null)
this.Res.write(ctx.SHOUTS().getText() + ':' + '
');
}
exitLine(ctx) {
this.Res.write("
");
}
}
After the required function calls, we make our HtmlChatListener to
extend ChatListener. The interesting stuff starts at line 12.
The ctx argument is an instance of a specific class context for the
node that we are entering/exiting. So for enterName is NameContext,
for exitEmoticon is EmoticonContext, etc. This specific context will
have the proper elements for the rule, that would make possible to
easily access the respective tokens and sub-rules. For example,
NameContext will contain fields like WORD() and WHITESPACE();
CommandContext will contain fields like WHITESPACE(), SAYS() and
SHOUTS().
These functions, enter* and exit*, are called by the walker every
time the corresponding nodes are entered or exited while it's
traversing the AST that represents the program newline. A
listener allows you to execute some code, but it's important to
remember that you cannot stop the execution of the walker and the
execution of the functions.
On line 13, we start by printing a strong tag because we want the
name to be bold, then on exitName we take the text from the token
WORD and close the tag. Note that we ignore the WHITESPACE token,
nothing says that we have to show everything. In this case we could
have done everything either on the enter or exit function, since
nothing is happening in between them.
On the function exitEmoticon we simply transform the emoticon text
into an emoji character. We get the text of the whole rule because
there are no tokens defined for this parser rule. On enterCommand,
instead there could be any of two tokens SAYS or SHOUTS, so we check
which one is defined. And then we alter the following text, by
transforming in uppercase, if it's a SHOUT. Note that we close the
p tag at the exit of the line rule, because the command, semantically
speaking, alter all the text of the message.
All we have to do now is launch node, with node antlr.js, and point
our browser at its address, usually at http://localhost:1337/ and we
will be greeted with the following image.
Output of the HtmlChatListener
So all is good, we just have to add all the different listeners to
handle the rest of the language. Let's start with color and message.
15. Working with a Listener
We have seen how to start defining a listener. Now let's get serious
and see how to evolve in a complete, robust listener. Let's start by
adding support for color and checking the results of our hard work.
enterColor(ctx) {
var color = ctx.WORD().getText();
this.Res.write('');
}
exitColor(ctx) {
this.Res.write("");
}
exitMessage(ctx) {
this.Res.write(ctx.getText());
}
Output of the HtmlChatListener with errors
Except that it does not work. Or maybe it works too much: we are
writing some part of message twice ("this will work"): first when we
check the specific nodes, which are children of message, and then at
the end. We are also writing the emoticon out of order: that is
because we are writing that directly instead of waiting for the end
of message.
Luckily with JavaScript we can dynamically alter objects, so we can
take advantage of this fact to change the *Context object themselves.
enterColor(ctx) {
var color = ctx.WORD().getText();
ctx.text = '';
}
exitColor(ctx) {
ctx.text += ctx.message().text;
ctx.text += '';
}
exitEmoticon(ctx) {
var emoticon = ctx.getText();
if(emoticon == ':-)' || emoticon == ':)')
{
ctx.text = "";
}
if(emoticon == ':-(' || emoticon == ':(')
{
ctx.text = "";
}
}
exitMessage(ctx) {
var text = '';
for (var index = 0; index < ctx.children.length; index++ ) {
if(ctx.children[index].text != null)
text += ctx.children[index].text;
else
text += ctx.children[index].getText();
}
if(ctx.parentCtx instanceof ChatParser.LineContext == false)
{
ctx.text = text;
}
else
{
this.Res.write(text);
this.Res.write("");
}
}
Only the modified parts are shown in the snippet above. We add a text
field to every node that transforms its text, and then at the exit of
every message we print the text if it is the primary message, the one
that is directly child of the line rule. If it is a message, that is
also a child of color, we add the text field to the node we are
exiting and let color print it. We check this on line 30, where we
look at the parent node to see if it's an instance of the object
LineContext. This is also further evidence of how each ctx argument
corresponds to the proper type.
Between lines 23 and 28 we can see another field of every node of the
generated tree: children, which obviously contains the children node.
You can observe that if a field text exists, we add it to the proper
variable, otherwise we use the usual function to get the text of the
node.
16. Solving Ambiguities with Semantic Predicates
So far we have seen how to build a parser for a chat language in
JavaScript. Let's continue working on this grammar but switch to
Python. Remember that all code is available in the repository. Before
that, we have to solve an annoying problem: the TEXT token. The
solution we have is terrible, and furthermore, if we tried to get
the text of the token, we would have to trim the edges, parentheses
or square brackets. So what can we do?
We can use a particular feature of ANTLR called semantic predicates.
As the name implies, they are expressions that produce a boolean
value. They selectively enable or disable the following rule and thus
permit to solve ambiguities. Another reason why they could be used is
to support different versions of the same language, for instance a
version with a new construct or an old without it.
Technically, they are part of the larger group of actions, that
allows to embed arbitrary code into the grammar. The downside is that
the grammar is no more language independent, since the code in the
action must be valid for the target language. For this reason,
usually it is considered a good idea to only use semantic predicates
when they cannot be avoided, and leave most of the code to the
visitor/listener.
link : '[' TEXT ']' '(' TEXT ')';
TEXT : {self._input.LA(-1) == ord('[') or self._input.LA(-1) == ord('(')}? ~[\])]+ ;
We restored link to its original formulation, but we added a semantic
predicate to the TEXT token, written inside curly brackets and
followed by a question mark. We use self._input.LA(-1) to check the
character before the current one, if this character is a square
bracket or the open parenthesis, we activate the TEXT token. It is
important to repeat that this must be valid code in our target
language, since it is going to end up in the generated Lexer or
Parser, in our case in ChatLexer.py.
This matters not just for the syntax itself, but also because
different targets might have different fields or methods, for
instance LA returns an int in python, so we have to convert the char
to a int.
Let's look at the equivalent form in other languages.
// C#. Notice that is .La and not .LA
TEXT : {_input.La(-1) == '[' || _input.La(-1) == '('}? ~[\])]+ ;
// Java
TEXT : {_input.LA(-1) == '[' || _input.LA(-1) == '('}? ~[\])]+ ;
// JavaScript
TEXT : {this._input.LA(-1) == '['.charCodeAt(0) || this._input.LA(-1) == '('.charCodeAt(0)}? ~[\])]+ ;
If you want to test for the preceding token, you can use the
_input.LT(-1), but you can only do that for parser rules. For
example, if you want to enable the mention rule only if preceded by a
WHITESPACE token.
// C#
mention: {_input.Lt(-1).Type == WHITESPACE}? '@' WORD ;
// Java
mention: {_input.LT(-1).getType() == WHITESPACE}? '@' WORD ;
// Python
mention: {self._input.LT(-1).text == ' '}? '@' WORD ;
// JavaScript
mention: {this._input.LT(-1).text == ' '}? '@' WORD ;
17. Continuing the Chat in Python
Before seeing the Python example, we must modify our grammar and put
the TEXT token before the WORD one. Otherwise ANTLR might assign the
incorrect token, in cases where the characters between parentheses or
brackets are all valid for WORD, for instance if it where [this]
(link).
Using ANTLR in python is not more difficult than with any other
platform, you just need to pay attention to the version of Python, 2
or 3.
antlr4 -Dlanguage=Python3 Chat.g4
And that's it. So when you have run the command, inside the directory
of your python project, there will be a newly generated parser and
a lexer. You may find interesting to look at ChatLexer.py and in
particular the function TEXT_sempred (sempred stands for semantic
predicate).
def TEXT_sempred(self, localctx:RuleContext, predIndex:int):
if predIndex == 0:
return self._input.LA(-1) == ord('[') or self._input.LA(-1) == ord('(')
You can see our predicate right in the code. This also means that you
have to check that the correct requirements are available to the
generated parser. For instance, you need to import somewhere in the
lexer any necessary library for the functions used in the predicate.
18. The Python Way of Working with a Listener
The main file of a Python project is very similar to a JavaScript
one, mutatis mutandis of course. That is to say we have to adapt
libraries and functions to the proper version for a different
language.
import sys
from antlr4 import *
from ChatLexer import ChatLexer
from ChatParser import ChatParser
from HtmlChatListener import HtmlChatListener
def main(argv):
input = FileStream(argv[1])
lexer = ChatLexer(input)
stream = CommonTokenStream(lexer)
parser = ChatParser(stream)
tree = parser.chat()
output = open("output.html","w")
htmlChat = HtmlChatListener(output)
walker = ParseTreeWalker()
walker.walk(htmlChat, tree)
output.close()
if __name__ == '__main__':
main(sys.argv)
We have also changed the input and output to become files, this
avoids the need to launch a server in Python. This also solves the
issue that the default terminal in Windows supports only some
characters (i.e., not Unicode).
import sys
from antlr4 import *
from ChatParser import ChatParser
from ChatListener import ChatListener
class HtmlChatListener(ChatListener) :
def __init__(self, output):
self.output = output
self.output.write('')
def enterName(self, ctx:ChatParser.NameContext) :
self.output.write("")
def exitName(self, ctx:ChatParser.NameContext) :
self.output.write(ctx.WORD().getText())
self.output.write(" ")
def enterColor(self, ctx:ChatParser.ColorContext) :
color = ctx.WORD().getText()
ctx.text = ''
def exitColor(self, ctx:ChatParser.ColorContext):
ctx.text += ctx.message().text
ctx.text += ''
def exitEmoticon(self, ctx:ChatParser.EmoticonContext) :
emoticon = ctx.getText()
if emoticon == ':-)' or emoticon == ':)' :
ctx.text = ""
if emoticon == ':-(' or emoticon == ':(' :
ctx.text = ""
def enterLink(self, ctx:ChatParser.LinkContext):
ctx.text = '%s' % (ctx.TEXT()[1], (ctx.TEXT()[0]))
def exitMessage(self, ctx:ChatParser.MessageContext):
text = ''
for child in ctx.children:
if hasattr(child, 'text'):
text += child.text
else:
text += child.getText()
if isinstance(ctx.parentCtx, ChatParser.LineContext) is False:
ctx.text = text
else:
self.output.write(text)
self.output.write("")
def enterCommand(self, ctx:ChatParser.CommandContext):
if ctx.SAYS() is not None :
self.output.write(ctx.SAYS().getText() + ':' + '')
if ctx.SHOUTS() is not None :
self.output.write(ctx.SHOUTS().getText() + ':' + '
')
def exitChat(self, ctx:ChatParser.ChatContext):
self.output.write("")
Apart from lines 35-36, in which we introduce support for links,
there is nothing new. Though you might notice that Python syntax is
cleaner and, while having dynamic typing, it is not loosely typed as
JavaScript. The different types of *Context objects are explicitly
written out. If only Python tools were as easy to use as the language
itself. But of course we cannot just fly over python like this, so we
also introduce testing.
19. Testing with Python
While Visual Studio Code has a very nice extension for Python that
also supports unit testing, we are going to use the command line for
the sake of compatibility.
python3 -m unittest discover -s . -p ChatTests.py
That's how you run the tests, but before that, we have to write them.
Actually, even before that, we have to write an ErrorListener to
manage errors that we could find. While we could simply read the text
outputted by the default error listener, there is an advantage in
using our own implementation, namely that we can control more easily
what happens.
import sys
from antlr4 import *
from ChatParser import ChatParser
from ChatListener import ChatListener
from antlr4.error.ErrorListener import *
import io
class ChatErrorListener(ErrorListener):
def __init__(self, output):
self.output = output
self._symbol = ''
def syntaxError(self, recognizer, offendingSymbol, line, column, msg, e):
self.output.write(msg)
self._symbol = offendingSymbol.text
@property
def symbol(self):
return self._symbol
Our class derives from ErrorListener and we simply have to implement
syntaxError. Although we also add a property symbol to easily check
which symbol might have caused an error.
from antlr4 import *
from ChatLexer import ChatLexer
from ChatParser import ChatParser
from HtmlChatListener import HtmlChatListener
from ChatErrorListener import ChatErrorListener
import unittest
import io
class TestChatParser(unittest.TestCase):
def setup(self, text):
lexer = ChatLexer(InputStream(text))
stream = CommonTokenStream(lexer)
parser = ChatParser(stream)
self.output = io.StringIO()
self.error = io.StringIO()
parser.removeErrorListeners()
errorListener = ChatErrorListener(self.error)
parser.addErrorListener(errorListener)
self.errorListener = errorListener
return parser
def test_valid_name(self):
parser = self.setup("John ")
tree = parser.name()
htmlChat = HtmlChatListener(self.output)
walker = ParseTreeWalker()
walker.walk(htmlChat, tree)
# let's check that there aren't any symbols in errorListener
self.assertEqual(len(self.errorListener.symbol), 0)
def test_invalid_name(self):
parser = self.setup("Joh-")
tree = parser.name()
htmlChat = HtmlChatListener(self.output)
walker = ParseTreeWalker()
walker.walk(htmlChat, tree)
# let's check the symbol in errorListener
self.assertEqual(self.errorListener.symbol, '-')
if __name__ == '__main__':
unittest.main()
The setup method is used to ensure that everything is properly set;
on lines 19-21 we setup also our ChatErrorListener, but first we
remove the default one, otherwise it would still output errors on the
standard output. We are listening to errors in the parser, but we
could also catch errors generated by the lexer. It depends on what
you want to test. You may want to check both.
The two proper test methods check for a valid and an invalid name.
The checks are linked to the property symbol, that we have previously
defined, if it's empty everything is fine, otherwise it contains the
symbol that created the error. Notice that on line 28, there is a
space at the end of the string, because we have defined the rule name
to end with a WHITESPACE token.
20. Parsing Markup
ANTLR can parse many things, including binary data, in that case
tokens are made up of non-printable characters. But a more common
problem is parsing markup languages such as XML or HTML. Markup is
also a useful format to adopt for your own creations, because it
allows to mix unstructured text content with structured annotations.
They fundamentally represent a form of smart document, containing
both text and structured data. The technical term that describes them
is island languages. The term is not restricted to include only
markup languages, and it is sometimes a matter of perspective.
For example, you may have to build a parser that only deal with
preprocessor directives. In that case, you have to find a way to
distinguish proper code from directives, which obeys different rules.
In such an example the preprocessor directives might be considered an
island language, a separate language surrounded by meaningless (for
parsing purposes) text.
In any case, the problem for parsing such languages is that there is
a lot of text that we do not actually have to parse, but we cannot
ignore or discard, because the text contains useful information for
the user and it is a structural part of the document. The solution is
lexical modes, a way to parse structured content inside a larger sea
of free text.
21. Lexical Modes
We are going to see how to use lexical modes, by starting with a new
grammar.
lexer grammar MarkupLexer;
OPEN : '[' -> pushMode(BBCODE) ;
TEXT : ~('[')+ ;
// Parsing content inside tags
mode BBCODE;
CLOSE : ']' -> popMode ;
SLASH : '/' ;
EQUALS : '=' ;
STRING : '"' .*? '"' ;
ID : LETTERS+ ;
WS : [ \t\r\n] -> skip ;
fragment LETTERS : [a-zA-Z] ;
Looking at the first line you could notice a difference: we are
defining a lexer grammar, instead of the usual (combined) grammar.
You simply cannot define a lexical mode together with a parser
grammar. You can use lexical modes only in a lexer grammar, not in a
combined grammar. The rest is not surprising, as you can see, we are
defining a sort of BBCode markup, with tags delimited by square
brackets.
On lines 3, 7 and 9 you will find basically all that you need to know
about lexical modes. You define one or more tokens that can delimit
the different modes and activate them.
The default mode is already implicitly defined, if you need to define
yours you simply use mode followed by a name. Other than for markup
languages, lexical modes are typically used to deal with string
interpolation. That is when a string literal can contain more than
simple text, for instance arbitrary expressions.
When we used combined grammar, we could define tokens implicitly:
that is what happened when we used a string like '=' in a parser
rule. Now that we are using separate lexer and parser grammars we
cannot do that. That means that every single token has to be defined
explicitly. So we have definitions like SLASH or EQUALS which
typically could just be directly used in a parser rule. The
concept is simple: in the lexer grammar, we need to define all
tokens, because they cannot be defined later in the parser grammar.
22. Parser Grammars
We look at the other side of a lexer grammar, so to speak.
parser grammar MarkupParser;
options { tokenVocab=MarkupLexer; }
file : element* ;
attribute : ID '=' STRING ;
content : TEXT ;
element : (content | tag) ;
tag : '[' ID attribute? ']' element* '[' '/' ID ']' ;
On the first line we define a parser grammar. Since the tokens we
need are defined in the lexer grammar, we need to use an option to
say to ANTLR where it can find them. This is not necessary in
combined grammars, since the tokens are defined in the same file.
There are many other options available, in the documentation.
There is almost nothing else to add, except that we define a content
rule so that we can manage more easily the text that we find later in
the program.
I just want to say that, as you can see, we do not need to explicitly
use the tokens every time (e.g., SLASH), but instead we can use the
corresponding text (e.g., '/').
ANTLR will automatically transform the text in the corresponding
token, but this can happen only if they are already defined. In
short, it is as if we had written:
tag : OPEN ID attribute? CLOSE element* OPEN SLASH ID CLOSE ;
But we could not have used the implicit way, if we had not already
explicitly defined them in the lexer grammar. Another way to look at
this is: when we define a combined grammar, ANTLR defines for us all
the tokens that we have not explicitly defined ourselves. When we
need to use a separate lexer and a parser grammar, we have to define
explicitly every token ourselves. Once we have done that, we can use
them in every way we want.
Before moving to actual Java code, let's see the parse tree for a
sample input.
Sample parse tree of the Markup parser
You can easily notice that the element rule is sort of transparent:
where you would expect to find it there is always going to be a tag
or content. So why did we define it? There are two advantages: avoid
repetition in our grammar and simplify managing the results of the
parsing.
We avoid repetition because if we did not have the element rule, we
should repeat (content|tag) everywhere it is used. What if one day we
add a new type of element? In addition to that, it simplifies the
processing of the parse tree, because it makes easy to act upon both
tag and content, given that you can use their common ancestor
(element).
Advanced
ANTLR FAQ Table of Contents
Your ANTLR Questions Answered
A collection of answers to common issues that you encounter and
useful patterns that you can use when creating ANTLR parsers.
We also included a cheatsheet for ANTLR that you can use as an handy
reference.
Learn More About the ANTLR FAQ
In this section we deepen our understanding of ANTLR. We will look at
more complex examples and situations we may have to handle in our
parsing adventures. We will learn how to perform more advanced
testing, to catch more bugs and ensure a better quality for our code.
We will see what a visitor is and how to use it. Finally, we will see
how to deal with expressions and the complexity they bring.
You can come back to this section when you need to deal with complex
parsing problems.
23. The Markup Project in Java
You can follow the instructions in Java Setup or just copy the
antlr-java folder of the companion repository. Once the file
build.gradle is properly configured, this is how you build and
execute the application.
# use gradle to build the project
./gradlew compileJava
# if you are not using an IDE
# and you have defined the fatJar task as in the repository
./gradlew fatJar
java -jar .\build\libs\markup-example-gradle-all.jar
As you can see, it is not any different from any typical Gradle
project, although it is indeed more complicated than a typical
JavaScript or Python project. Of course, if you use an IDE, you do
not need to do anything different from your typical workflow.
24. The Main App.java
We are going to see how to write a typical ANTLR application in Java.
package me.tomassetti.examples.MarkupParser;
import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.tree.*;
public class App
{
public static void main( String[] args )
{
CharStream inputStream = CharStreams.fromString(
"I would like to [b][i]emphasize[/i][/b] this and [u]underline [b]that[/b][/u] ." +
"Let's not forget to quote: [quote author=\"John\"]You're wrong![/quote]");
MarkupLexer markupLexer = new MarkupLexer(inputStream);
CommonTokenStream commonTokenStream = new CommonTokenStream(markupLexer);
MarkupParser markupParser = new MarkupParser(commonTokenStream);
MarkupParser.FileContext fileContext = markupParser.file();
MarkupVisitor visitor = new MarkupVisitor();
visitor.visit(fileContext);
}
}
There is a small surprise regarding the inputStream variable. Instead
of using an [ANTLR]InputStream class we are using a CharStream one.
This difference applies to Java and C# and it is due to Unicode
support. The gist is that previous version of the runtimes for these
languages supported only partially Unicode, so to avoid any
surprising change in behavior the old [ANTLR]InputStream was
deprecated. You can read more on the official ANTLR documentation.
Apart from this change, at this point the main java file should not
come as a surprise: the only new development is the visitor. Of
course, there are the obvious little differences in the names of the
ANTLR classes and such. This time we are building a visitor, whose
main advantage is the chance to control the flow of the program.
While we are still dealing with text, we do not want to display it,
we want to transform it from pseudo-BBCode to pseudo-Markdown.
25. Transforming Code with ANTLR
The first issue to deal with our translation from pseudo-BBCode to
pseudo-Markdown is a design decision. Our two languages are different
and frankly neither of the two original ones is that well designed.
BBCode was created as a safety precaution, to make possible to
disallow the use of HTML but give some of its power to users.
Markdown was created to be an easy to read and write format, that
could be translated into HTML. So they both mimic HTML, and you can
actually use HTML in a Markdown document. Let's start looking into
how messy a real conversion would be.
package me.tomassetti.examples.MarkupParser;
import org.antlr.v4.runtime.*;
import org.antlr.v4.runtime.misc.*;
import org.antlr.v4.runtime.tree.*;
public class MarkupVisitor extends MarkupParserBaseVisitor
{
@Override
public String visitFile(MarkupParser.FileContext context)
{
visitChildren(context);
System.out.println("");
return null;
}
@Override
public String visitContent(MarkupParser.ContentContext context)
{
System.out.print(context.TEXT().getText());
return visitChildren(context);
}
}
The first version of our visitor prints all the text and ignore all
the tags.
You can see how to control the flow, either by calling visitChildren,
or any other visit* function, and deciding what to return. We just
need to override the methods that we want to change. Otherwise, the
default implementation would just do like visitContent, on line 24,
it will visit the children nodes and allow the visitor to continue.
Just like for a listener, the argument is the proper context type. If
you want to stop the visitor, just return null as on line 16.
26. Joy and Pain of Transforming Code
Transforming code, even at a very simple level, comes with some
complications. Let's start easy with some basic visitor methods.
@Override
public String visitContent(MarkupParser.ContentContext context)
{
return context.getText();
}
@Override
public String visitElement(MarkupParser.ElementContext context)
{
if(context.parent instanceof MarkupParser.FileContext)
{
if(context.content() != null)
System.out.print(visitContent(context.content()));
if(context.tag() != null)
System.out.print(visitTag(context.tag()));
}
return null;
}
Before looking at the main method, let's look at the supporting ones.
Foremost, we have changed visitContent by making it return its text
instead of printing it. Second, we have overridden the visitElement
so that it prints the text of its child, but only if it is a top
element, and not inside a tag. In both cases, it achieves this by
calling the proper visit* method. It knows which one to call because
it checks if it actually has a tag or content node.
@Override
public String visitTag(MarkupParser.TagContext context)
{
String text = "";
String startDelimiter = "", endDelimiter = "";
String id = context.ID(0).getText();
switch(id)
{
case "b":
startDelimiter = endDelimiter = "**";
break;
case "u":
startDelimiter = endDelimiter = "*";
break;
case "quote":
String attribute = context.attribute().STRING().getText();
attribute = attribute.substring(1,attribute.length()-1);
startDelimiter = System.lineSeparator() + "> ";
endDelimiter = System.lineSeparator() + "> " + System.lineSeparator() + "> - "
+ attribute + System.lineSeparator();
break;
}
text += startDelimiter;
for (MarkupParser.ElementContext node: context.element())
{
if(node.tag() != null)
text += visitTag(node.tag());
if(node.content() != null)
text += visitContent(node.content());
}
text += endDelimiter;
return text;
}
VisitTag contains more code than every other method, because it can
also contain other elements, including other tags that have to be
managed themselves, and thus they cannot be simply printed. We save
the content of the ID on line 7, of course we don't need to check
that the corresponding end tag matches, because the parser will
ensure that, as long as the input is well formed.
The first complication starts with at lines 14-15: as it often
happens when transforming a language in a different one, there is not
a perfect correspondence between the two. While BBCode tries to be a
smarter and safer replacement for HTML, Markdown wants to accomplish
the same objective of HTML, to create a structured document. So
BBCode has an underline tag, while Markdown does not.
So we have to make a decision
Do we want to discard the information, or directly print HTML, or
something else? We choose something else and instead convert the
underline to an italic. That might seem completely arbitrary, and
indeed there is an element of choice in this decision. But the
conversion forces us to lose some information, and both are used for
emphasis, so we choose the closer thing in the new language.
The following case, on lines 18-22, forces us to make another choice.
We can't maintain the information about the author of the quote in a
structured way, so we choose to print the information in a way that
will make sense to a human reader.
On lines 28-34 we do our magic: we visit the children and gather
their text, then we close with the endDelimiter. Finally we return
the text that we have created.
That's how the visitor works
1. every top element visits each child
+ if it's a content node, it directly returns the text
+ if it's a tag, it setups the correct delimiters and then it
checks its children. It repeats step 2 for each child and
then it returns the gathered text
2. it prints the returned text
It is obviously a simple example, but it shows how you can have great
freedom in managing the visitor once you have launched it. Together
with the patterns that we have seen at the beginning of this section,
you can see all of the options: to return null to stop the visit, to
return children to continue, to return something to perform an action
ordered at a higher level of the tree.
27. Advanced Testing
The use of lexical modes permits handling the parsing of island
languages, but it complicates testing.
We are not going to show MarkupErrorListener.java because we did not
change it; if you need it, you can see it on the repository.
You can run the tests by using the following command.
./gradlew test
Now we are going to look at the tests code. We are skipping the setup
part, because that also is obvious, we just copy the process seen on
the main file, but we simply add our error listener to intercept the
errors.
// private variables inside the class AppTest
private MarkupErrorListener errorListener;
private MarkupLexer markupLexer;
@Test
public void testText()
{
MarkupParser parser = setup("anything in here");
MarkupParser.ContentContext context = parser.content();
assertEquals("",this.errorListener.getSymbol());
}
@Test
public void testInvalidText()
{
MarkupParser parser = setup("[anything in here");
MarkupParser.ContentContext context = parser.content();
// note that this.errorListener.symbol could be empty
// when ANTLR doesn't recognize the token or there is no error.
// In such cases check the output of errorListener
assertEquals("[",this.errorListener.getSymbol());
}
@Test
public void testWrongMode()
{
MarkupParser parser = setup("author=\"john\"");
MarkupParser.AttributeContext context = parser.attribute();
TokenStream ts = parser.getTokenStream();
assertEquals(MarkupLexer.DEFAULT_MODE, markupLexer._mode);
assertEquals(MarkupLexer.TEXT,ts.get(0).getType());
assertEquals("author=\"john\"",this.errorListener.getSymbol());
}
@Test
public void testAttribute()
{
MarkupParser parser = setup("author=\"john\"");
// we have to manually push the correct mode
this.markupLexer.pushMode(MarkupLexer.BBCODE);
MarkupParser.AttributeContext context = parser.attribute();
TokenStream ts = parser.getTokenStream();
assertEquals(MarkupLexer.ID,ts.get(0).getType());
assertEquals(MarkupLexer.EQUALS,ts.get(1).getType());
assertEquals(MarkupLexer.STRING,ts.get(2).getType());
assertEquals("",this.errorListener.getSymbol());
}
@Test
public void testInvalidAttribute()
{
MarkupParser parser = setup("author=/\"john\"");
// we have to manually push the correct mode
this.markupLexer.pushMode(MarkupLexer.BBCODE);
MarkupParser.AttributeContext context = parser.attribute();
assertEquals("/",this.errorListener.getSymbol());
}
The first two methods are exactly as before, we simply check that
there are no errors, or that there is the correct one because the
input itself is erroneous. On lines 36-38 things start to get
interesting: the issue is that by testing the rules one by one, we do
not give the chance to the parser to switch automatically to the
correct mode. So it remains always on the DEFAULT_MODE, which in our
case makes everything looks like TEXT. This obviously makes the
correct parsing of an attribute impossible.
The same lines also show how you can check the current mode that you
are in, and the exact type of the tokens that are found by the
parser, which we use to confirm that indeed all is wrong in this
case.
While we could use a string of text to trigger the correct mode each
time, that would make testing intertwined with several pieces of
code, which is a no-no. So the solution is seen on line 46: we
trigger the correct mode manually. Once you have done that, you can
see that our attribute is recognized correctly.
28. Dealing with Expressions
So far we have written simple parser rules, now we are going to see
one of the most challenging parts in analyzing a real (programming)
language: expressions. While rules for statements are usually larger,
they are quite simple to deal with: you just need to write a rule
that encapsulate the structure with all the different optional parts.
For instance, a for statement can include all other kinds of
statements, but we can simply include them with something like
statement*. An expression, instead, can be combined in many different
ways.
An expression usually contains other expressions. For example, the
typical binary expression is composed of an expression on the left,
an operator in the middle and another expression on the right. This
can lead to ambiguities. Think, for example, at the expression 5 + 3
* 2, for ANTLR this expression is ambiguous because there are two
ways to parse it. It could either parse it as 5 + (3 * 2) or (5 +3) *
2.
Until this moment, we have avoided the problem simply because markup
constructs surround the object on which they are applied. So there is
not ambiguity in choosing which one to apply first: it is the most
external. Imagine if this expression was written as:
5
3
2
That would make obvious to ANTLR how to parse it.
These types of rules are called left-recursive rules. You might say:
just parse whatever comes first. The problem with that is semantic:
the addition comes first, but we know that multiplications have
precedence over additions. Traditionally, the way to solve this
problem was to create a complex cascade of specific expressions like
this:
expression : addition;
addition : multiplication ('+' multiplication)* ;
multiplication : atom ('*' atom)* ;
atom : NUMBER ;
This way ANTLR would have known to search first for a number, then
for multiplications and finally for additions. This is cumbersome and
also counterintuitive, because the last expression is the first to be
actually recognized. Luckily ANTLR4 can create a similar structure
automatically, so we can use a much more natural syntax.
expression : expression '*' expression
| expression '+' expression
| NUMBER
;
In practice ANTLR consider the order in which we defined the
alternatives to decide the precedence. By writing the rule in this
way, we are telling to ANTLR that the multiplication takes precedence
over the addition.
29. Parsing Spreadsheets
Now we are prepared to create our last application, in C#. We are
going to build the parser of an Excel-like application. In practice,
we want to manage the expressions you write in the cells of a
spreadsheet.
grammar Spreadsheet;
expression : '(' expression ')' #parenthesisExp
| expression (ASTERISK|SLASH) expression #mulDivExp
| expression (PLUS|MINUS) expression #addSubExp
| expression '^' expression #powerExp
| NAME '(' expression ')' #functionExp
| NUMBER #numericAtomExp
| ID #idAtomExp
;
fragment LETTER : [a-zA-Z] ;
fragment DIGIT : [0-9] ;
ASTERISK : '*' ;
SLASH : '/' ;
PLUS : '+' ;
MINUS : '-' ;
ID : LETTER DIGIT ;
NAME : LETTER+ ;
NUMBER : DIGIT+ ('.' DIGIT+)? ;
WHITESPACE : ' ' -> skip;
With all the knowledge you have acquired so far everything should be
clear, except for possibly three things:
1. why the parentheses are there,
2. what's the stuff on the right,
3. that thing on line 6.
Parentheses to change the operator precedence
The parentheses come first because their only role is to give the
user a way to override the precedence of operator, if they need to do
so. This graphical representation of the AST should make it clear.
The things on the right are labels, they are used to make ANTLR
generate specific functions for the visitor or listener. So there
will be a VisitFunctionExp, a VisitPowerExp, etc. This makes possible
to avoid the use of a giant visitor for the expression rule.
The expression relative to exponentiation is different because there
are two possible ways to act, to group them, when you meet two
sequential expressions of the same type. The first one is to execute
the one on the left first and then the one on the right, the second
one is the inverse: this is called associativity. Usually the one
that you want to use is left-associativity, which is the default
option. Nonetheless exponentiation is right-associative, so we have
to signal this to ANTLR.
Another way to look at this is: if there are two expressions of the
same type, which one has the precedence: the left one or the right
one? Again, an image is worth a thousand words.
Associativity of an expression
We also have support for functions, alphanumeric variables that
represents cells and real numbers.
30. The Spreadsheet Project in C#
You just need to follow the C# Setup: to install a nuget package for
the runtime and an ANTLR4 extension for Visual Studio. The extension
will automatically generate everything whenever you build your
project: parser, listener and/or visitor.
Notice that there are two small differences between the code for a
project using the extension and one using the Java tool. These are
noted in the README for the C# project at the repository.
After you have done that, you can also add grammar files just by
using the usual menu Add -> New Item. Do exactly that to create a
grammar called Spreadsheet.g4 and put in it the grammar we have just
created. Now let's see the main Program.cs.
using System;
using Antlr4.Runtime;
namespace AntlrTutorial
{
class Program
{
static void Main(string[] args)
{
string input = "log(10 + A1 * 35 + (5.4 - 7.4))";
ICharStream inputStream = CharStreams.fromString(input);
SpreadsheetLexer spreadsheetLexer = new SpreadsheetLexer(inputStream);
CommonTokenStream commonTokenStream = new CommonTokenStream(spreadsheetLexer);
SpreadsheetParser spreadsheetParser = new SpreadsheetParser(commonTokenStream);
SpreadsheetParser.ExpressionContext expressionContext = spreadsheetParser.expression();
SpreadsheetVisitor visitor = new SpreadsheetVisitor();
Console.WriteLine(visitor.Visit(expressionContext));
}
}
}
There is nothing to say, apart from that, of course, you have to pay
attention to yet another slight variation in the naming of things:
pay attention to the casing and interfaces. For instance,
ICharStream, in the C# program, was CharStream in the Java program.
Also you can notice that, this time, we output on the screen the
result of our visitor, instead of writing the result on a file.
31. Excel is Doomed
We are going to take a look at our visitor for the Spreadsheet
project.
public class DoubleSpreadsheetVisitor : SpreadsheetBaseVisitor
{
private static DataRepository data = new DataRepository();
public override double VisitNumericAtomExp(SpreadsheetParser.NumericAtomExpContext context)
{
return double.Parse(context.NUMBER().GetText(), System.Globalization.CultureInfo.InvariantCulture);
}
public override double VisitIdAtomExp(SpreadsheetParser.IdAtomExpContext context)
{
String id = context.ID().GetText();
return data[id];
}
public override double VisitParenthesisExp(SpreadsheetParser.ParenthesisExpContext context)
{
return Visit(context.expression());
}
public override double VisitMulDivExp(SpreadsheetParser.MulDivExpContext context)
{
double left = Visit(context.expression(0));
double right = Visit(context.expression(1));
double result = 0;
if (context.ASTERISK() != null)
result = left * right;
if (context.SLASH() != null)
result = left / right;
return result;
}
[..]
public override double VisitFunctionExp(SpreadsheetParser.FunctionExpContext context)
{
String name = context.NAME().GetText();
double result = 0;
switch(name)
{
case "sqrt":
result = Math.Sqrt(Visit(context.expression()));
break;
case "log":
result = Math.Log10(Visit(context.expression()));
break;
}
return result;
}
}
VisitNumeric and VisitIdAtom return the actual numbers that are
represented either by the literal number or the variable. In a real
scenario DataRepository would contain methods to access the data in
the proper cell, but in our example is just a Dictionary with some
keys and numbers. The other methods actually work in the same way:
they visit/call the containing expression(s). The only difference is
what they do with the results.
Some perform an operation on the result, the binary operations
combine two results in the proper way and finally VisitParenthesisExp
just reports the result higher on the chain. Math is simple, when it
is done by a computer.
32. Testing Everything
Up until now, we have only tested the parser rules, that is to say we
have tested only if we have created the correct rule to parse our
input. Now we are also going to test the visitor functions. This is
the ideal chance because our visitor returns values that we can check
individually. In other occasions, for instance if your visitor prints
something to the screen, you may want to rewrite the visitor to write
on a stream. Then, at testing time, you can easily capture the
output.
We are not going to show SpreadsheetErrorListener.cs because it is
the same as the previous one we have already seen; if you need it you
can see it on the repository.
To perform unit testing with C#, you need to create a specific
project inside the solution. You can choose different formats, we opt
for the xUnit version. To run them there is an aptly named section
"TEST" on the menu bar in Visual Studio or the command dotnet test on
the command line.
[Fact]
public void testExpressionPow()
{
setup("5^3^2");
PowerExpContext context = parser.expression() as PowerExpContext;
CommonTokenStream ts = (CommonTokenStream)parser.InputStream;
Assert.Equal(SpreadsheetLexer.NUMBER, ts.Get(0).Type);
Assert.Equal(SpreadsheetLexer.T__2, ts.Get(1).Type);
Assert.Equal(SpreadsheetLexer.NUMBER, ts.Get(2).Type);
Assert.Equal(SpreadsheetLexer.T__2, ts.Get(3).Type);
Assert.Equal(SpreadsheetLexer.NUMBER, ts.Get(4).Type);
}
[Fact]
public void testVisitPowerExp()
{
setup("4^3^2");
PowerExpContext context = parser.expression() as PowerExpContext;
SpreadsheetVisitor visitor = new SpreadsheetVisitor();
double result = visitor.VisitPowerExp(context);
Assert.Equal(double.Parse("262144"), result);
}
[..]
[Fact]
public void testWrongVisitFunctionExp()
{
setup("logga(100)");
FunctionExpContext context = parser.expression() as FunctionExpContext;
SpreadsheetVisitor visitor = new SpreadsheetVisitor();
double result = visitor.VisitFunctionExp(context);
CommonTokenStream ts = (CommonTokenStream)parser.InputStream;
Assert.Equal(SpreadsheetLexer.NAME, ts.Get(0).Type);
Assert.Equal(null, errorListener.Symbol);
Assert.Equal(0, result);
}
[Fact]
public void testCompleteExp()
{
setup("log(5+6*7/8)");
ExpressionContext context = parser.expression();
SpreadsheetVisitor visitor = new SpreadsheetVisitor();
double result = visitor.Visit(context);
Assert.Equal("1.0107238653917732", result.ToString(System.Globalization.CultureInfo.GetCultureInfo("en-US").NumberFormat));
}
The first test function is similar to the ones we have already seen;
it checks that the correct tokens are selected. On line 11 and 13 you
may be surprised to see that weird token type, this happens because
we didn't explicitly create one for the '^' symbol so one got
automatically created for us. If you need you can see all the tokens
by looking at the *.tokens file generated by ANTLR.
On line 25 we visit our test node and get the results, that we check
on line 27. It is all very simple because our visitor is simple.
While unit testing should always be easy and made up of small parts
it really cannot be easier than this.
The only thing to pay attention to is related to the format of the
number; it is not a problem here, but look at line 59, where we test
the result of a whole expression. There we need to make sure that the
correct format is selected, because different countries use different
symbols as the decimal mark.
There are some things that depend on the cultural context
If your computer was already set to the American English Culture this
would not be necessary, but to guarantee the correct testing results
for everybody, we have to specify it. Keep that in mind if you are
testing things that are culture-dependent: such as grouping of
digits, temperatures, etc.
On line 44-46 you see that when we check for the wrong function the
parser actually works. That is because indeed logga is syntactically
valid as a function name, but it is not semantically correct. The
function logga does not exist, so our program does not know what to
do with it. So when we visit it we get 0 as a result. As you recall
this was our choice: since we initialize the result to 0 and we do
not have a default case in VisitFunctionExp. So if there is no
function the result remains 0. A possible alternative could be to
throw an exception.
Final Remarks
In this section we see tips and tricks that never came up in our
example, but can be useful in your programs. We suggest more
resources you may find useful if you want to know more about ANTLR,
both the practice and the theory, or you need to deal with the most
complex problems.
33. Tips and Tricks
Let's see a few tricks that could be useful from time to time. These
were never needed in our examples, but they have been quite useful in
other scenarios.
Catchall Rule
The first one is the ANY lexer rule. This is simply a rule in the
following format.
ANY : . ;
This is a catchall rule that should be put at the end of your
grammar. It matches any character that didn't find its place during
the parsing. So creating this rule can help you during development,
when your grammar has still many holes that could cause distracting
error messages. It's even useful during production, when it acts as a
canary in the mines. If it shows up in your program you know that
something is wrong.
Channels
There is also something that we have not talked about: channels.
Their use case is usually handling comments. You do not really want
to check for comments inside every of your statements or expressions,
so you usually throw them away with -> skip. But there are some cases
where you may want to preserve them, for instance if you are
translating a program in another language. When this happens, you use
channels. There is already one called HIDDEN that you can use, but
you can declare more of them at the top of your lexer grammar.
channels { UNIQUENAME }
// and you use them this way
COMMENTS : '//' ~[rn]+ -> channel(UNIQUENAME) ;
Rule Element Labels
There is another use of labels other than to distinguish among
different cases of the same rule. They can be used to give a specific
name to a common rule or parts of a rule. The format is label=rule,
to be used inside another rule.
expression : left=expression (ASTERISK|SLASH) right=expression ;
This way left and right would become fields in the ExpressionContext
nodes. And instead of using context.expression(0), you could refer to
the same entity using context.left.
Problematic Tokens
In many real languages a few symbols are reused in different ways,
which in some cases may lead to ambiguities. A common problematic
example are the angle brackets, used both for bitshift expression and
to delimit parameterized types.
// bitshift expression, it assigns to x the value of y shifted by three bits
x = y >> 3;
// parameterized types, it define x as a list of dictionaries
List> x;
The natural way of defining the bitshift operator token is as a
single double angle brackets, >>. But this might lead to confusing a
nested parameterized definition with the bitshift operator, for
instance in the second example shown up here. While a simple way of
solving the problem would be using semantic predicates, an excessive
number of them would slow down the parsing phase. The solution is to
avoid defining the bitshift operator token and instead using the
angle brackets twice in the parser rule, so that the parser itself
can choose the best candidate for every occasion.
// from this
RIGHT_SHIFT : '>>';
expression : ID RIGHT_SHIFT NUMBER;
// to this
expression : ID SHIFT SHIFT NUMBER;
34. Conclusions
Using ANTLR Like a Professional
Using ANTLR Like a Professional
A complete video course on parsing and ANTLR, that will teach you how
to build parser for everything from programming languages to data
formats.
Now, available as an improved II Edition.
Show Me the Course
We have learned a lot today:
* what a lexer and a parser are
* how to create lexer and parser rules
* how to use ANTLR to generate parsers in Java, C#, Python and
JavaScript
* the fundamental kinds of problems you will encounter parsing and
how to solve them
* how to understand errors
* how to test your parsers
That's all you need to know to use ANTLR on your own. And I mean
literally, you may want to know more, but now you have solid basis to
explore on your own.
Where to look if you need more information about ANTLR:
* On this very website there is whole category dedicated to ANTLR.
* The official ANTLR website is a good starting point to know the
general status of the project, the specialized development tools
and related project, like StringTemplate
* The ANTLR documentation on GitHub; especially useful are the
information on targets and how to setup it on different languages
.
* The ANTLR API; it's related to the Java version, so there might
be some differences in other languages, but it's the best place
where to settle your doubts about the inner workings of this
tool.
* For the very interested in the science behind ANTLR4, there is an
academic paper: Adaptive LL(*) Parsing: The Power of Dynamic
Analysis
* The Definitive ANTLR 4 Reference, by the man itself, Terence Parr
, the creator of ANTLR. The resource you need if you want to know
everything about ANTLR and a good deal about parsing languages in
general.
Also the book is only place where you can find and answer to question
like these:
ANTLR v4 is the result of a minor detour (twenty-five years) I
took in graduate
school. I guess I'm going to have to change my motto slightly.
Why program by hand in five days what you can spend twenty-five
years of your
life automating?
If instead you decide you could use some help with your projects
involving ANTLR, you can also use our ANTLR Consulting Services.
We would like to thank Bernard Kaiflin for having revised the
document and helped us improving it.
We would like to thank: Brasilio Castilho, Andy Nicholas, grz0,
scinod for having spotted errors and typos in the article.
We worked quite hard to build the largest tutorial on ANTLR: the
mega-tutorial! A post over 14.000 words long, or more than 70 pages,
to try answering all your questions about ANTLR. Missing something?
Contact us and let us now, we are here to help.
Categories
* ANTLR
* Application modernization
* Code processing
* Consulting
* Domain specific languages
* Editors
* Jetbrains MPS
* Language design
* Language Engineering
* Language Workbenches
* Miscellany
* Model driven development
* Open-source
* Parsing
* Software Development
More on ANTLR
[viktor-jakovlev-H0vuplqoX0c-unsplash-1-scaled]
Strumenta Talk: in conversation with Rob Wunderlich
21 April 2022
[services]
Getting Started With ANTLR in PHP
05 April 2022
[logo] Strumenta Tools to solve complex problems
Privacy Policy
P.IVA 11817320010
Company Information
* Strumenta
* Strumenta
* Federico Tomassetti
* Federico Tomassetti
We use cookies on our website to give you the most relevant
experience by remembering your preferences and repeat visits. By
clicking "Accept", you consent to the use of ALL the cookies.
Cookie settingsACCEPT
Privacy & Cookies Policy
Close
Privacy Overview
This website uses cookies to improve your experience while you
navigate through the website. Out of these cookies, the cookies that
are categorized as necessary are stored on your browser as they are
essential for the working of basic functionalities of the website. We
also use third-party cookies that help us analyze and understand how
you use this website. These cookies will be stored in your browser
only with your consent. You also have the option to opt-out of these
cookies. But opting out of some of these cookies may have an effect
on your browsing experience.
Necessary
[*] Necessary
Always Enabled
Necessary cookies are absolutely essential for the website to
function properly. This category only includes cookies that ensures
basic functionalities and security features of the website. These
cookies do not store any personal information.
Non-necessary
[*] Non-necessary
Any cookies that may not be particularly necessary for the website to
function and is used specifically to collect user personal data via
analytics, ads, other embedded contents are termed as non-necessary
cookies. It is mandatory to procure user consent prior to running
these cookies on your website.
SAVE & ACCEPT
*