Quelle arch.xml Sprache: XML

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE api-answers PUBLIC "-//NetBeans//DTD Arch Answers//EN" "../nbbuild/antsrc/org/netbeans/nbbuild/Arch.dtd" [
  <!ENTITY api-questions SYSTEM "../nbbuild/antsrc/org/netbeans/nbbuild/Arch-api-questions.xml">
]>

<api-answers
  question-version="1.29"
  author="mmetelka@netbeans.org"
>

  &api-questions;


<answer id="arch-what">
Lexer module provides token lists for various
text inputs. Token lists can either be flat or they can form
tree token hierarchies if any language embedding is present.
Tokens
</answer>


<answer id="arch-overall">
The lexer module defines
<api name="LexerAPI" group="java" type="export" category="official"/>
providing access to sequence of tokens for various input sources.
<br/>
An <b>API entry point</b> is
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenHierarchy.html">TokenHierarchy</a>
class with its static methods that provide its instance for the given input source.

<h2>Input Sources</h2>
<p>
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenHierarchy.html">TokenHierarchy</a>
    can be created for immutable input sources (
<a href="@JDK@@JDKMODULE_JAVA_BASE@/java/lang/CharSequence.html">CharSequence</a>
    or
<a href="@JDK@@JDKMODULE_JAVA_BASE@/java/io/Reader.html">java.io.Reader</a>
    ) or for mutable input sources (typically
<a href="@JDK@@JDKMODULE_JAVA_DESKTOP@/javax/swing/text/Document.html">javax.swing.text.Document</a>
    ).
    <br/>
    For mutable input source the lexer framework updates the tokens in the token hierarchy automatically
    with subsequent changes to the underlying text input.
    The tokens of the hierarchy always reflect the text of the input at the given time.
</p>

<h2>TokenSequence and Token</h2>
<p>
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenHierarchy.html#tokenSequence()">TokenHierarchy.tokenSequence()</a>
   allows to iterate over a list of
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/Token.html">Token</a>
    instances.
    <br/>
    The token carries a token identification
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenId.html">TokenId</a>
    (returned by
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/Token.html#id()">Token.id()</a>
    ) and a text (aka token body) represented as
<a href="@JDK@@JDKMODULE_JAVA_BASE@/java/lang/CharSequence.html">CharSequence</a>
    (returned by
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/Token.html#text()">Token.text()</a>
    ).
    <br/>
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenUtilities.html">TokenUtilities</a>
    contains many useful methods related to operations with the token's text such as
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenUtilities.html#equals(java.lang.CharSequence,java.lang.Object)">TokenUtilities.equals(CharSequence text, Object o)</a>,
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenUtilities.html#startsWith(java.lang.CharSequence,java.lang.CharSequence)">TokenUtilities.startsWith(CharSequence text, CharSequence prefix)</a>,
   etc.
    <br/>
   It is also possible to debug the text of the token (replace special chars by escapes) by
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenUtilities.html#debugText(java.lang.CharSequence)">TokenUtilities.equals(CharSequence text)</a>.
    <br/>
    A typical token also carries offset of its occurrence in the input text.
</p>

<h2>Flyweight Tokens</h2>
<p>
    As there are many token occurrences where the token text is the same for all
    or many occurrences
    (e.g. java keywords, operators or a single-space whitespace) the memory consumption
    can be decreased considerably by allowing the creation of <b>flyweight token</b> instances
    i.e. just one token instance is used for all the token's occurrences
    in all the inputs.
    <br/>
    Flyweight tokens can be determined by
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/Token.html#isFlyweight()">Token.isFlyweight()</a>.
    <br/>
    The flyweight tokens do not carry a valid offset (their internal offset is -1).
    <br/>
    Therefore
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenSequence.html">TokenSequence</a>
    is used for iteration through the tokens (instead of a regular iterator) and it provides
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenSequence.html#offset()">TokenSequence.offset()</a>
    which returns the proper offset even when positioned over a flyweight token.
    <br/>
    When holding a reference to the token's instance its offset can also be determined by
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/Token.html#offset(org.netbeans.api.lexer.TokenHierarchy)">Token.offset(TokenHierarchy tokenHierarchy)</a>.
    The <code>tokenHierarchy</code> parameter should be always <code>null</code> and it will be used
    for the token hierarchy snapshot support in future releases.
    <br/>
    For flyweight tokens the
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/Token.html#offset(org.netbeans.api.lexer.TokenHierarchy)">Token.offset(TokenHierarchy tokenHierarchy)</a>
    returns -1 and for regular tokens it gives the same value like
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenSequence.html#offset()">TokenSequence.offset()</a>.
</p>

<p>
    There may be applications where the flyweight tokens use could be problematic.
    For example if a parser would like to use token instances
    in a parse tree nodes to determine the nodes' boundaries then the flyweight tokens
    would always return offset -1 so the positions of the parse tree nodes
    could not generally be determined from the tokens only.
    <br/>
    Therefore there is a possibility to de-flyweight a token by using
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenSequence.html#offsetToken()">TokenSequence.offsetToken()</a>
    which checks the current token
    and if it's flyweight then it replaces it with a non-flyweight token instance
    with a valid offset and with the same properties as the original flyweight token.
</p>

<h2>TokenId and Language</h2>
<p>
    Token is identified by its id represented by
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenId.html">TokenId</a>
    interface. Token ids for a language are typically implemented as java enums (extensions of
<a href="@JDK@@JDKMODULE_JAVA_BASE@/java/lang/Enum.html">Enum</a>
    ) but it's not mandatory.
    <br/>
    All token ids for the given language are described by
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/Language.html">Language</a>.
    <br/>
    Each token id may belong
    to one or more token categories that allow to better operate
    tokens of the same type (e.g. keywords or operators).
    <br/>
    Each token id may define its primary category
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenId.html#primaryCategory()">TokenId.primaryCategory()</a>
    and
<a href="@org-netbeans-modules-lexer@/org/netbeans/spi/lexer/LanguageHierarchy.html#createTokenCategories()">LanguageHierarchy.createTokenCategories()</a>
    may provide additional categories for the token ids for the given language.
    <br/>
    Each language description has a mandatory mime-type specification
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/Language.html#mimeType()">Language.mimeType()</a>
    <br/>
    Although it's a bit non-related information it brings many benefits
    because with the mime-type the language can be accompanied
    with an arbitrary sort of settings (e.g. syntax coloring information etc.).
</p>

<h2>LanguageHierarchy, Lexer, LexerInput and TokenFactory</h2>
<p>
    SPI providers wishing to provide a
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/Language.html">Language</a>
    first need to define its SPI counterpart
<a href="@org-netbeans-modules-lexer@/org/netbeans/spi/lexer/LanguageHierarchy.html">LanguageHierarchy</a>.
    It mainly needs to define token ids in
<a href="@org-netbeans-modules-lexer@/org/netbeans/spi/lexer/LanguageHierarchy.html#createTokenIds()">LanguageHierarchy.createTokenIds()</a>
    and lexer in
<a href="@org-netbeans-modules-lexer@/org/netbeans/spi/lexer/LanguageHierarchy.html#createLexer(org.netbeans.spi.lexer.LexerRestartInfo)">
    LanguageHierarchy.createLexer(LexerInput lexerInput, TokenFactory tokenFactory, Object state, LanguagePath languagePath, InputAttributes inputAttributes)</a>.
    <br/>
<a href="@org-netbeans-modules-lexer@/org/netbeans/spi/lexer/Lexer.html">Lexer</a>
    reads characters from
<a href="@org-netbeans-modules-lexer@/org/netbeans/spi/lexer/LexerInput.html">LexerInput</a>
    and breaks the text into tokens.
    <br/>
    Tokens are produced by using methods of
<a href="@org-netbeans-modules-lexer@/org/netbeans/spi/lexer/TokenFactory.html">TokenFactory</a>.
    <br/>
    As a per-token memory consumption is critical the
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/Token.html">Token</a>
    does not have any counterpart in SPI. However the framework prevents instantiation
    of any other token classes except those contained in the lexer module's implementation.
</p>

<h2>Language Embedding</h2>
<p>
    With language embedding the flat list of tokens becomes in fact a tree-like hierarchy
    represented by the
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenHierarchy.html">TokenHierarchy</a>
    class. Each token can potentially be broken into a sequence of embedded tokens.
    <br/>The
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenSequence.html#embedded()">TokenSequence.embedded()</a>
    method can be called to obtain the embedded tokens (when positioned on the branch token).
    <br/>
    There are two ways of specifying what language is embedded in a token. The language
    can either be specified explicitly (hardcoded) in the
<a href="@org-netbeans-modules-lexer@/org/netbeans/spi/lexer/LanguageHierarchy.html#embedding(org.netbeans.api.lexer.Token,org.netbeans.api.lexer.LanguagePath,org.netbeans.api.lexer.InputAttributes)">LanguageHierarchy.embedding()</a>
    method or there can be a
<a href="@org-netbeans-modules-lexer@/org/netbeans/spi/lexer/LanguageProvider.html">LanguageProvider</a>
    registered in the default Lookup, which will create a
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/Language.html">Language</a>
    for the embedded language.
    <br/>
    There is no limit on the depth of a language hierarchy and there can be as many embedded languages
    as needed.
    <br/>
    In SPI the language embedding is represented by
<a href="@org-netbeans-modules-lexer@/org/netbeans/spi/lexer/LanguageEmbedding.html">LanguageEmbedding</a>.
</p>

</answer>


<answer id="arch-usecases">



<h1>
API Usecases
</h1>

<h2>
Obtaining of token hierarchy for various inputs.
</h2>
The
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/TokenHierarchy.html">TokenHierarchy</a>
is an entry point into Lexer API
and it represents the given input in terms of tokens.
<pre>
    String text = "public void m() { }";
    TokenHierarchy hi = TokenHierarchy.create(text, JavaLanguage.description());
</pre>

<br/>
Token hierarchy for swing documents must be operated under read/write document's lock.
<pre>
    document.readLock();
    try {
        TokenHierarchy hi = TokenHierarchy.get(document);
        ... // explore tokens etc.
    } finally {
        document.readUnlock();
    }
</pre>

<h2>
Obtaining and iterating token sequence over particular swing document from the given offset.
</h2>
The tokens cover the whole document and it's possible to iterate either forward or backward.
<br/>
Each token can contain language embedding that can also be explored by the token sequence.
The language embedding covers the whole text of the token (there can be few characters
skipped at the begining an end of the branch token).

<pre>
    document.readLock();
    try {
        TokenHierarchy hi = TokenHierarchy.get(document);
        TokenSequence ts = hi.tokenSequence();
        // If necessary move ts to the requested offset
        ts.move(offset);
        while (ts.moveNext()) {
            Token t = ts.token();
            if (t.id() == ...) { ... }
            if (TokenUtilities.equals(t.text(), "mytext")) { ... }
            if (ts.offset() == ...) { ... }

            // Possibly retrieve embedded token sequence
            TokenSequence embedded = ts.embedded();
            if (embedded != null) { // Token has a valid language embedding
                ...
            }
        }
    } finally {
        document.readUnlock();
    }
</pre>

<br/>
Typical clients:
<ul>
    <li>Editor's painting code doing syntax coloring
        <code>org.netbeans.modules.lexer.editorbridge.LexerLayer</code> in <i>lexer/editorbridge</i> module.
    </li>
    <li>Brace matching code searching for matching brace in forward/backward direction.</li>
    <li>Code completion's quick check whether caret is located inside comment token.</li>
    <li>Parser constructing a parse tree iterating through the tokens in forward direction.</li>
</ul>

<h2>
Using language path of the token sequence
</h2>
For the given token sequence the client may check whether it's a top level
token sequence in the token hierarchy or whether it's embedded at which level
it's embedded and what are the parent languages.
<br/>
Each token can contain language embedding that can also be explored by the token sequence.
The language embedding covers the whole text of the token (there can be few characters
skipped at the begining an end of the branch token).

<pre>
    TokenSequence ts = ...
    LanguagePath lp = ts.languagePath();
    if (lp.size() > 1) { ... } // This is embedded token sequence
    if (lp.topLanguage() == JavaLanguage.description()) { ... } // top-level language of the token hierarchy
    String mimePath = lp.mimePath();
    Object setting-value = some-settings.getSetting(mimePath, setting-name);
</pre>

<h2>
Extra information about the input
</h2>
The
<a href="@org-netbeans-modules-lexer@/org/netbeans/api/lexer/InputAttributes.html">InputAttributes</a>
class may carry extra information about the text input on which the token hierarchy
is being created. For example there can be information about the version of the language
that the input represents and the lexer may be written to recognize multiple versions
of the language. It should suffice to do the versioning through a simple integer:
<pre>
public class MyLexer implements Lexer<MyTokenId> {

    private final int version;

    ...

    public MyLexer(LexerInput input, TokenFactory<MyTokenId> tokenFactory, Object state,
    LanguagePath languagePath, InputAttributes inputAttributes) {
        ...

        Integer ver = (inputAttributes != null)
                ? (Integer)inputAttributes.getValue(languagePath, "version")
                : null;
        this.version = (ver != null) ? ver.intValue() : 1; // Use version 1 if not specified explicitly
    }

    public Token<MyTokenId> nextToken() {
        ...
        if (recognized-assert-keyword) {
            return (version >= 4) { // "assert" recognized as keyword since version 4
                ? keyword(MyTokenId.ASSERT)
                : identifier();
        }
        ...
    }
    ...
}
</pre>

The client will then use the following code:
<pre>
    InputAttributes attrs = new InputAttributes();
    // The "true" means global value i.e. for any occurrence of the MyLanguage including embeddings
    attrs.setValue(MyLanguage.description(), "version", Integer.valueOf(3), true);
    TokenHierarchy hi = TokenHierarchy.create(text, false, SimpleLanguage.description(), null, attrs);
    ...
</pre>

<h2>
Filtering out unnecessary tokens
</h2>
Filtering is only possible for immutable inputs (e.g. String or Reader).
<pre>
    Set<MyTokenId> skipIds = EnumSet.of(MyTokenId.COMMENT, MyTokenId.WHITESPACE);
    TokenHierarchy tokenHierarchy = TokenHierarchy.create(inputText, false,
        MyLanguage.description(), skipIds, null);
    ...
</pre>

<br/>
Typical clients:
<ul>
    <li>Parser constructing a parse tree. It is not interested
        in the comment and whitespace tokens so these tokens do not need
        to be constructed at all.
    </li>
</ul>



<h1>
SPI Usecases
</h1>

<h2>
Providing language description and lexer.
</h2>

Token ids should be defined as enums. For example
<code>org.netbeans.lib.lexer.test.simple.SimpleTokenId</code> can be copied
or the following example from
<code>org.netbeans.modules.lexer.editorbridge.calc.lang.CalcTokenId</code>.
<br/>
The static <code>language()</code> method returns the language describing the token ids.
<pre>
public enum CalcTokenId implements TokenId {

    WHITESPACE(null, "whitespace"),
    SL_COMMENT(null, "comment"),
    ML_COMMENT(null, "comment"),
    E("e", "keyword"),
    PI("pi", "keyword"),
    IDENTIFIER(null, null),
    INT_LITERAL(null, "number"),
    FLOAT_LITERAL(null, "number"),
    PLUS("+", "operator"),
    MINUS("-", "operator"),
    STAR("*", "operator"),
    SLASH("/", "operator"),
    LPAREN("(", "separator"),
    RPAREN(")", "separator"),
    ERROR(null, "error"),
    ML_COMMENT_INCOMPLETE(null, "comment");

    private final String fixedText;

    private final String primaryCategory;

    private CalcTokenId(String fixedText, String primaryCategory) {
        this.fixedText = fixedText;
        this.primaryCategory = primaryCategory;
    }

    public String fixedText() {
        return fixedText;
    }

    public String primaryCategory() {
        return primaryCategory;
    }

    private static final Language<CalcTokenId> language = new LanguageHierarchy<CalcTokenId>() {
        <code>@Override</code>
        protected Collection<CalcTokenId> createTokenIds() {
            return EnumSet.allOf(CalcTokenId.class);
        }

        <code>@Override</code>
        protected Map<String,Collection<CalcTokenId>> createTokenCategories() {
            Map<String,Collection<CalcTokenId>> cats = new HashMap<String,Collection<CalcTokenId>>();

            // Incomplete literals
            cats.put("incomplete", EnumSet.of(CalcTokenId.ML_COMMENT_INCOMPLETE));
            // Additional literals being a lexical error
            cats.put("error", EnumSet.of(CalcTokenId.ML_COMMENT_INCOMPLETE));

            return cats;
        }

        <code>@Override</code>
        protected Lexer<CalcTokenId> createLexer(LexerRestartInfo<CalcTokenId> info) {
            return new CalcLexer(info);
        }

        <code>@Override</code>
        protected String mimeType() {
            return "text/x-calc";
        }

    }.language();

    public static final Language<CalcTokenId> language() {
        return language;
    }

}
</pre>

Note that it is not needed to publish the underlying <code>LanguageHierarchy</code> extension.

<br/>
Lexer example:
<pre>
public final class CalcLexer implements Lexer<CalcTokenId> {

    private static final int EOF = LexerInput.EOF;

    private static final Map<String,CalcTokenId> keywords = new HashMap<String,CalcTokenId>();
    static {
        keywords.put(CalcTokenId.E.fixedText(), CalcTokenId.E);
        keywords.put(CalcTokenId.PI.fixedText(), CalcTokenId.PI);
    }

    private LexerInput input;

    private TokenFactory<CalcTokenId> tokenFactory;

    CalcLexer(LexerRestartInfo<CalcTokenId> info) {
        this.input = info.input();
        this.tokenFactory = info.tokenFactory();
        assert (info.state() == null); // passed argument always null
    }

    public Token<CalcTokenId> nextToken() {
        while (true) {
            int ch = input.read();
            switch (ch) {
                case '+':
                    return token(CalcTokenId.PLUS);

                case '-':
                    return token(CalcTokenId.MINUS);

                case '*':
                    return token(CalcTokenId.STAR);

                case '/':
                    switch (input.read()) {
                        case '/': // in single-line comment
                            while (true)
                                switch (input.read()) {
                                    case '\r': input.consumeNewline();
                                    case '\n':
                                    case EOF:
                                        return token(CalcTokenId.SL_COMMENT);
                                }
                        case '*': // in multi-line comment
                            while (true) {
                                ch = input.read();
                                while (ch == '*') {
                                    ch = input.read();
                                    if (ch == '/')
                                        return token(CalcTokenId.ML_COMMENT);
                                    else if (ch == EOF)
                                        return token(CalcTokenId.ML_COMMENT_INCOMPLETE);
                                }
                                if (ch == EOF)
                                    return token(CalcTokenId.ML_COMMENT_INCOMPLETE);
                            }
                    }
                    input.backup(1);
                    return token(CalcTokenId.SLASH);

                case '(':
                    return token(CalcTokenId.LPAREN);

                case ')':
                    return token(CalcTokenId.RPAREN);

                case '0': case '1': case '2': case '3': case '4':
                case '5': case '6': case '7': case '8': case '9':
                case '.':
                    return finishIntOrFloatLiteral(ch);

                case EOF:
                    return null;

                default:
                    if (Character.isWhitespace((char)ch)) {
                        ch = input.read();
                        while (ch != EOF && Character.isWhitespace((char)ch)) {
                            ch = input.read();
                        }
                        input.backup(1);
                        return token(CalcTokenId.WHITESPACE);
                    }

                    if (Character.isLetter((char)ch)) { // identifier or keyword
                        while (true) {
                            if (ch == EOF || !Character.isLetter((char)ch)) {
                                input.backup(1); // backup the extra char (or EOF)
                                // Check for keywords
                                CalcTokenId id = keywords.get(input.readText());
                                if (id == null) {
                                    id = CalcTokenId.IDENTIFIER;
                                }
                                return token(id);
                            }
                            ch = input.read(); // read next char
                        }
                    }

                    return token(CalcTokenId.ERROR);
            }
        }
    }

    public Object state() {
        return null;
    }

    private Token<CalcTokenId> finishIntOrFloatLiteral(int ch) {
        boolean floatLiteral = false;
        boolean inExponent = false;
        while (true) {
            switch (ch) {
                case '.':
                    if (floatLiteral) {
                        return token(CalcTokenId.FLOAT_LITERAL);
                    } else {
                        floatLiteral = true;
                    }
                    break;
                case '0': case '1': case '2': case '3': case '4':
                case '5': case '6': case '7': case '8': case '9':
                    break;
                case 'e': case 'E': // exponent part
                    if (inExponent) {
                        return token(CalcTokenId.FLOAT_LITERAL);
                    } else {
                        floatLiteral = true;
                        inExponent = true;
                    }
                    break;
                default:
                    input.backup(1);
                    return token(floatLiteral ? CalcTokenId.FLOAT_LITERAL
                            : CalcTokenId.INT_LITERAL);
            }
            ch = input.read();
        }
    }

    private Token<CalcTokenId> token(CalcTokenId id) {
        return (id.fixedText() != null)
            ? tokenFactory.getFlyweightToken(id, id.fixedText())
            : tokenFactory.createToken(id);
    }

}
</pre>
<p>
    The classes containing token ids and the language description should be
    part of an API. The lexer should only be part of the implementation.
</p>

<h2>
Providing language embedding.
</h2>

The embedding may be provided statically
in the <code>LanguageHierarchy.embedding()</code>
see e.g. <code>org.netbeans.lib.lexer.test.simple.SimpleLanguage</code>.

<p>
    Or it may be provided dynamically through the xml layer
    by using a file in "Editors/language-mime-type/languagesEmbeddingMap" folder
    named by the token-id's name containing target mime-type and initial and ending skip lengths:
</p>
<pre>
    <folder name="Editors">
        <folder name="text">
            <folder name="x-outer-language">
                <folder name="languagesEmbeddingMap">
                    <file name="WORD"><![CDATA[text/x-inner-language,1,2]]>
                    </file>
                </folder>
            </folder>
        </folder>
    </folder>
</pre>

</answer>


<answer id="arch-quality">
The lexer module is completely unit-testable.
<br/>
Besides of tests for its own correctness it also contains support
for testing of correctness of lexers from SPI providers
by using <code>org.netbeans.lib.lexer.test.TestRandomModify</code> class.
<br/>
The main testing method for the lexer correctnes is token-by-token comparing
of the updated token sequence with a batch-lexed token sequence for the same input.
</answer>


<answer id="arch-time">
The present implementation is stable but there are few missing implementations
and other things to be considered:
<ul>
    <li>Dynamic language embedding binding through xml layer.</li>
    <li>CharPreprocessor servicing and tests.</li>
    <li>Token hierarchy for Reader.</li>
    <li>TokenFactory.createBranchToken() impl.</li>
    <li>Providing JavaCC and Antlr support.</li>
    <li>Support for token positions (may add API).</li>
</ul>
</answer>


<answer id="compat-i18n">
Yes.
</answer>


<answer id="compat-standards">
Compatible with standards.
</answer>


<answer id="compat-version">
Yes.
</answer>


<answer id="dep-jre">
JDK1.4 and higher can be used.
</answer>


<answer id="dep-jrejdk">
JRE is sufficient.
</answer>


<answer id="dep-nb">
    <defaultanswer generate='here'/>
</answer>


<answer id="dep-non-nb">
No other projects.
</answer>


<answer id="dep-platform">
All platforms.
</answer>


<answer id="deploy-dependencies">
<pre>
OpenIDE-Module-Module-Dependencies: org.netbeans.modules.lexer/2 > @SPECIFICATION-VERSION@
</pre>
</answer>


<answer id="deploy-jar">
No additional files.
</answer>


<answer id="deploy-nbm">
Yes.
</answer>


<answer id="deploy-packages">
Yes, where appropriate.
</answer>


<answer id="deploy-shared">
Anywhere.
</answer>


<answer id="exec-classloader">
No.
</answer>


<answer id="exec-component">
No.
</answer>


<answer id="exec-introspection">
No.
</answer>


<answer id="exec-privateaccess">
No.
</answer>


<answer id="exec-process">
No.
</answer>


<answer id="exec-property">
    <api type="export" group="logger" name="org.netbeans.lib.lexer.TokenHierarchyOperation" category="friend">
        <code>FINE</code> level lists lexer changes made in tokens both at the root level
        and embedded levels of the token hierarchy after each document modification.
        <br/>
        <code>FINER</code> level in addition will also check the whole token hierarchy
         for internal consistency after each modification.
    </api>
    <api type="export" group="logger" name="org.netbeans.lib.lexer.TokenList" category="friend">
        <code>FINE</code> level forces lexer to perform more thorough and strict checks
        in certain situations so this is useful mainly for tests.
        Lookahead and state information is generated even for batch-lexed inputs which allows
        easier checking of incremental algorithm correctness (fixing of token list after modification).
        There are also some additional checks performed
        that should verify correctness of the framework and the SPI implementation
        classes being used (for example when flyweight tokens are created the text
        passed to the token factory is compared to the text in the lexer input).
    </api>
</answer>


<answer id="exec-reflection">
No.
</answer>


<answer id="exec-threading">
Use of token hierarchies for mutable input sources
must adhere to the locking mechanisms for the input sources themselves.
<br/>
For example accessing token hierarchy for swing document
requires read/write locking of document prior accessing token hierarchy.
</answer>


<answer id="format-clipboard">
No clipboard support.
</answer>


<answer id="format-dnd">
No D&D.
</answer>


<answer id="format-types">
No files read or written to the disk.
</answer>


<answer id="lookup-lookup">
No
</answer>


<answer id="lookup-register">
No.
</answer>


<answer id="lookup-remove">
No.
</answer>


<answer id="perf-exit">
No.
</answer>


<answer id="perf-huge_dialogs">
No.
</answer>


<answer id="perf-limit">
No practical limits.
</answer>


<answer id="perf-mem">
Memory consumption is critical for created tokens because there can be thousands
of tokens per typical document. Thus there are several basic token types:
<ul>
    <li>DefaultToken: 24 bytes </li>
    <li>StringToken: 32 bytes (but only used for flyweight tokens)</li>
    <li>PrepToken: 32 bytes plus text storage size (but only used
        for tokens where character preprocessing was necessary)
    </li>
</ul>
</answer>


<answer id="perf-menus">
No.
</answer>


<answer id="perf-progress">
All the tasks should be granularized.
Both batch and incremental lexing is done lazily as clients ask for tokens.
<br/>
The only potential long-running task is relexing of a very long portion of documents
e.g. if someone would type '/*' at the begining of java document
without any comments - the whole document turns into unclosed comment.
<br/>
This typically isn't a problem unless the very long token does not need to be lexed
several times (the original support without permanent tokens had to lex the token
upon each request).
<br/>
The lexer framework further helps to improve the situation by introducing
token validation which attempts to validate the token by checking
whether the typed character may really affect the token
or whether it's just necessary to fix the original token's length.

</answer>


<answer id="perf-scale">
On a typical machine the framework is able to produce about 370,000 tokens
of a text input with 1 million characters in less than 0.5 second.
</answer>


<answer id="perf-spi">
The token change listeners implementations should be written to execute quickly.
For complex tasks they should reschedule its work into another thread.
</answer>


<answer id="perf-startup">
No.
</answer>


<answer id="perf-wakeup">
No.
</answer>


<answer id="resources-file">
No.
</answer>


<answer id="resources-layer">
No.
</answer>


<answer id="resources-mask">
No.
</answer>


<answer id="resources-read">
No.
</answer>


<answer id="security-grant">
No.
</answer>


<answer id="security-policy">
No.
</answer>


<answer id="exec-ant-tasks">
No.
</answer>


<answer id="arch-where">
  <defaultanswer generate='here' />
</answer>


<answer id="compat-deprecation">
  <p>
   The current API completely replaces the original one therefore
   the major version of the module was increased from 1 to 2.
   <br/>
   There are no plans to deprecated any part of the present API
   and it should be evolved in a compatible way.
  </p>
</answer>


<answer id="resources-preferences">
  <p>
   No.
  </p>
</answer>

</api-answers>

Messung V0.5 in Prozent

¤ Dauer der Verarbeitung: 0.35 Sekunden (vorverarbeitet am 2026-09-27) ¤

Wurzel

Suchen

PVS Prover

Isabelle Prover

NIST Cobol Testsuite

Cephes Mathematical Library

Vienna Development Method

Haftungshinweis

Die Informationen auf dieser Webseite wurden nach bestem Wissen sorgfältig zusammengestellt. Es wird jedoch weder Vollständigkeit, noch Richtigkeit, noch Qualität der bereit gestellten Informationen zugesichert.

Bemerkung:

Die farbliche Syntaxdarstellung und die Messung sind noch experimentell.