Book/tokenizer-Phpdoc专题

Tokenizer

简介
安装／配置
预定义常量
范例
PhpToken — The PhpToken class
- PhpToken::__construct — Returns a new PhpToken object
- PhpToken::getTokenName — Returns the name of the token.
- PhpToken::is — Tells whether the token is of given kind.
- PhpToken::isIgnorable — Tells whether the token would be ignored by the PHP parser.
- PhpToken::__toString — Returns the textual content of the token.
- PhpToken::tokenize — Splits given source into PHP tokens, represented by PhpToken objects.
Tokenizer 函数
- token_get_all — 将提供的源码按 PHP 标记进行分割
- token_name — 获取提供的 PHP 解析器代号的符号名称

简介

This class provides an alternative to token_get_all. While the function returns tokens either as a single-character string, or an array with a token ID, token text and line number, PhpToken::tokenize normalizes all tokens into PhpToken objects, which makes code operating on tokens more memory efficient and readable.

类摘要

PhpToken

class PhpToken {

/* 属性 */

public int $id ;

public string $text ;

public int $line ;

public int $pos ;

/* 方法 */

final public __construct ( int $id , string $text [, int $line = -1 [, int $pos = -1 ]] )

public stringnull getTokenName ( void )

public bool is ( intstringarray $kind )

public bool isIgnorable ( void )

public string __toString ( void )

public static array tokenize ( string $code [, int $flags = 0 ] )

}

属性

id
One of the T_* constants, or an ASCII codepoint representing a single-char token.

text
The textual content of the token.

line
The starting line number (1-based) of the token.

pos
The starting position (0-based) in the tokenized string.

PhpToken::__construct

Returns a new PhpToken object

说明

final public PhpToken::__construct ( int $id , string $text [, int $line = -1 [, int $pos = -1 ]] )

Returns a new PhpToken object

参数

id
One of the T_* constants (see 解析器代号列表), or an ASCII codepoint representing a single-char token.

text
The textual content of the token.

line
The starting line number (1-based) of the token.

pos
The starting position (0-based) in the tokenized string.

返回值

Returns a new PhpToken instance.

参见

PhpToken::tokenize

PhpToken::getTokenName

Returns the name of the token.

说明

public stringnull PhpToken::getTokenName ( void )

Returns the name of the token.

参数

此函数没有参数。

返回值

An ASCII character for single-char tokens, or one of T_* constant names for known tokens (see 解析器代号列表), or null for unknown tokens.

范例

示例 #1 PhpToken::getTokenName example

<?php
// known token
$token = new PhpToken(T_ECHO, 'echo');
var_dump($token->getTokenName());   // -> string(6) "T_ECHO"

// single-char token
$token = new PhpToken(ord(';'), ';');
var_dump($token->getTokenName());   // -> string(1) ";"

// unknown token
$token = new PhpToken(10000 , "\0");
var_dump($token->getTokenName());   // -> NULL

参见

PhpToken::tokenize
token_name

PhpToken::is

Tells whether the token is of given kind.

说明

public bool PhpToken::is ( intstringarray $kind )

Tells whether the token is of given kind.

参数

kind
Either a single value to match the token's id or textual content, or an array thereof.

返回值

A boolean value whether the token is of given kind.

范例

示例 #1 PhpToken::is example

<?php
$token = new PhpToken(T_ECHO, 'echo');
var_dump($token->is(T_ECHO));        // -> bool(true)
var_dump($token->is('echo'));        // -> bool(true)
var_dump($token->is(T_FOREACH));     // -> bool(false)
var_dump($token->is('foreach'));     // -> bool(false)

示例 #2 Usage with array

<?php
function isClassType(PhpToken $token): bool {
    return $token->is([T_CLASS, T_INTERFACE, T_TRAIT]);
}

$interface = new PhpToken(T_INTERFACE, 'interface');
var_dump(isClassType($interface));   // -> bool(true)

$function = new PhpToken(T_FUNCTION, 'function');
var_dump(isClassType($function));    // -> bool(false)

参见

token_name

PhpToken::isIgnorable

Tells whether the token would be ignored by the PHP parser.

说明

public bool PhpToken::isIgnorable ( void )

Tells whether the token would be ignored by the PHP parser.

参数

此函数没有参数。

返回值

A boolean value whether the token would be ignored by the PHP parser (such as whitespace or comments).

范例

示例 #1 PhpToken::isIgnorable example

<?php
$echo = new PhpToken(T_ECHO, 'echo');
var_dump($echo->isIgnorable());   // -> bool(false)

$space = new PhpToken(T_WHITESPACE, ' ');
var_dump($space->isIgnorable());  // -> bool(true)

参见

PhpToken::tokenize

PhpToken::__toString

Returns the textual content of the token.

说明

public string PhpToken::__toString ( void )

Returns the textual content of the token.

参数

此函数没有参数。

返回值

A textual content of the token.

范例

示例 #1 PhpToken::__toString example

<?php
$token = new PhpToken(T_ECHO, 'echo');
echo $token;

以上例程会输出：

echo

参见

token_name

PhpToken::tokenize

Splits given source into PHP tokens, represented by PhpToken objects.

说明

public static array PhpToken::tokenize ( string $code [, int $flags = 0 ] )

Returns an array of PhpToken objects representing given code.

参数

code
The PHP source to parse.

flags
Valid flags:

TOKEN_PARSE - Recognises the ability to use reserved words in specific contexts.

返回值

An array of PHP tokens represented by instances of PhpToken or its descendants. This method returns static[] so that PhpToken can be seamlessly extended.

范例

示例 #1 PhpToken::tokenize example

<?php
$tokens = PhpToken::tokenize('<?php echo; ?>');

foreach ($tokens as $token) {
    echo "Line {$token->line}: {$token->getTokenName()} ('{$token->text}')", PHP_EOL;
}

以上例程会输出：

Line 1: T_OPEN_TAG ('<?php ')
Line 1: T_ECHO ('echo')
Line 1: ; (';')
Line 1: T_WHITESPACE (' ')
Line 1: T_CLOSE_TAG ('?>')

示例 #2 Extending PhpToken

<?php

class MyPhpToken extends PhpToken {
    public function getUpperText() {
        return strtoupper($this->text);
    }
}

$tokens = MyPhpToken::tokenize('<?php echo; ?>');
echo "'{$tokens[0]->getUpperText()}'";

以上例程会输出：

'<?PHP '

参见

token_get_all