Quelle Lexer.java

Sprache: JAVA

/*
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/

package org.netbeans.spi.lexer;

import org.netbeans.api.lexer.TokenId;
import org.netbeans.api.lexer.Token;

/**
* Lexer reads input characters from {@link LexerInput} and groups
* them into tokens.
* 
* The lexer delegates token creation
* to {@link TokenFactory#createToken(TokenId)}.
* Token factory instance should be given to the lexer in its constructor.
*
* 
* The lexer must be able to express its internal lexing
* state at token boundaries and it must be able
* to restart lexing from such state.
* 
* It is expected that if the input characters following the restart point
* would not change then the lexer will return the same tokens
* regardless whether it was restarted at the restart point
* or run from the input begining as a batch lexer.
* 
*
* 
* Testing of the lexers:
* 
* Testing of newly written lexers can be performed in several ways.
* The most simple way is to test batch lexing first
* (see e.g.
* <a href="https://github.com/apache/netbeans/tree/master/ide/lexer/test/unit/src/org/netbeans/lib/lexer/test/simple/SimpleLexerBatchTest.java">
* org.netbeans.lib.lexer.test.simple.SimpleLexerBatchTest</a> in lexer module tests).
* 
* Then an "incremental" behavior of the new lexer can be tested
* (see e.g. <a href="https://github.com/apache/netbeans/tree/master/ide/lexer/test/unit/src/org/netbeans/lib/lexer/test/simple/SimpleLexerIncTest.java">
* org.netbeans.lib.lexer.test.simple.SimpleLexerIncTest</a>).
* 
* Finally the lexer can be tested by random tests that randomly insert and remove
* characters from the document
* (see e.g. <a href="https://github.com/apache/netbeans/tree/master/ide/lexer/test/unit/src/org/netbeans/lib/lexer/test/simple/SimpleLexerRandomTest.java">
* org.netbeans.lib.lexer.test.simple.SimpleLexerRandomTest</a>).
* 
* Once these tests pass the lexer can be considered stable.
* 
*
* @author Miloslav Metelka
* @version 1.00
*/

public interface Lexer<T extends TokenId> {

 /**
 * Return a token based on characters of the input
 * and possibly additional input properties.
 * 
 * Characters can be read by using
 * {@link LexerInput#read()} method. Once the lexer
 * knows that it has read enough characters to recognize
 * a token it calls
 * {@link TokenFactory#createToken(TokenId)}
 * to obtain an instance of a {@link Token} and then returns it.
 *
 * 
 * Note: Lexer must *not* return any other <code>Token</code> instances than
 * those obtained from the TokenFactory.
 * 
 *
 * 
 * The lexer is required to tokenize all the characters (except EOF)
 * provided by the {@link LexerInput} prior to returning null
 * from this method. Not doing so is treated
 * as malfunctioning of the lexer.
 * 
 *
 * @return token recognized by the lexer
 * or null if there are no more characters (available in the input) to be tokenized.
 * 
 * Return {@link TokenFactory#SKIP_TOKEN}
 * if the token should be skipped because of a token filter.
 *
 * @throws IllegalStateException if the token instance created by the lexer
 * was not created by the methods of TokenFactory (there is a common superclass
 * for those token implementations).
 * @throws IllegalStateException if this method returns null but not all
 * the characters of the lexer input were tokenized.
 */
 Token<T> nextToken();

 /**
 * This method is called by lexer's infrastructure
 * to return present lexer's state
 * once the lexer has recognized and returned a token.
 * 
 * In mutable environment this method is called after each recognized token
 * and its result is paired (together with token's lookahead) with the token
 * for later use - when lexer needs to be restarted at the token boundary.
 *
 * 
 * If the lexer is in no extra state (it is in a default state)
 * it should return <code>null</code>. Most lexers are in the default state
 * only at all the time.
 * 
 * If possible the non-default lexer states should be expressed
 * as small non-negative integers.
 * 
 * There is an optimization that shrinks the storage costs for small
 * <code>java.lang.Integer</code>s to single bytes.
 * 
 *
 * 
 * The returned value should not be tied to this particular lexer instance in any way.
 * Another lexer instance may be restarted from this state later.
 * 
 *
 * @return valid state object or null if the lexer is in a default state.
 */
 Object state();

 /**
 * Infrastructure calls this method when it no longer needs this lexer for lexing
 * so it becomes unused.
 * 
 * If lexer instances are cached and reused later
 * then this method should first release all the references that might cause
 * memory leaks and then add this unused lexer to the cache.
 */
 void release();

}

Messung V0.5 in Prozent

¤ Dauer der Verarbeitung: 0.9 Sekunden (vorverarbeitet am 2026-06-10) ¤

Wurzel

Suchen

Beweissystem der NASA

Beweissystem Isabelle

NIST Cobol Testsuite

Cephes Mathematical Library

Wiener Entwicklungsmethode

Haftungshinweis

Die Informationen auf dieser Webseite wurden nach bestem Wissen sorgfältig zusammengestellt. Es wird jedoch weder Vollständigkeit, noch Richtigkeit, noch Qualität der bereit gestellten Informationen zugesichert.

Bemerkung:

Die farbliche Syntaxdarstellung und die Messung sind noch experimentell.