Lex CDATA tags in chunks

Instead of using a single token (T_CDATA) for a CDATA tag the lexer now
uses 3 tokens:

1. T_CDATA_START
2. T_CDATA_BODY
3. T_CDATA_END

The T_CDATA_BODY token can occur multiple times and is turned into a
single value in the XML parser. This is similar to the way strings are
lexed.

By changing the way CDATA tags are lexed Oga can now lex CDATA tags
containing newlines when using an IO as input. For example, this would
previously fail:

    Oga.parse_xml(StringIO.new("<![CDATA[\nfoo]]>"))

Because IO input reads input per line the input for the lexer would be
as following:

    "<![CDATA[\n"
    "foo]]>"

Related issues: #93
This commit is contained in:
Yorick Peterse committed 2015-04-14 22:45:55 +02:00
1 parent 739e3b474c
commit 8acc7fc743
8 files changed
+153 -18

No files matched your search

+3 -1
View File
@@ -104,7 +104,9 @@ public class Lexer extends RubyObject
String id_advance_line = "advance_line";
String id_on_attribute = "on_attribute";
String id_on_attribute_ns = "on_attribute_ns";
String id_on_cdata = "on_cdata";
String id_on_cdata_start = "on_cdata_start";
String id_on_cdata_body = "on_cdata_body";
String id_on_cdata_end = "on_cdata_end";
String id_on_comment = "on_comment";
String id_on_doctype_end = "on_doctype_end";
String id_on_doctype_inline = "on_doctype_inline";