Class Token

A token including basic morphological and syntactic information. A token is similar to a word, but includes punctuation. Tokens have an zero-based index reflecting their position within their sentence. The morphological and syntactical features might be null (deepLemma, lemma, morphTag, pos, fnc, parent), or empty (children) if not requested or supported.

Hierarchy (View Summary)

Properties

charSpan children deepLemma feats fnc id idx lemma morphTag origCharSpan origText parent pos sentence subFnc text FEAT_LEMMA_INFO FEAT_NEGATED FEAT_UNKNOWN

Accessors

fullFnc

Methods

coverage depth filteredInOrder filteredPreOrder inOrder isContinuous isLeaf isNegated isRoot isUnknown leftChildren next offsetToken preOrder previous rigthChildren toIndentTreeString toSimpleIndentString toSimpleString toSimpleTreeString toString toStringWith toTreeString of

Properties

charSpan

charSpan: CharSpan

Cahracter span within the paragraph.

children

children: Token[] = []

Nodes that depend on this node, ordered by word-order; empty for leafs or if no tree-structure is in the analysis.

deepLemma

deepLemma: null | string

Lemma of the token e.g. bezpecny. null if not requested/supported.

feats

feats: Map<string, string>

Universal and custom features

fnc

fnc: null | UDep

Label of the dependency edge. Null if not requested/supported.

Readonlyid

id: string

ID of this node used to refer to it from other objects.

Readonlyidx

idx: number

Zero-based index of this node reflecting its word-order position within the sentence.

lemma

lemma: null | string

Simple lemma of the token, e.g. nejnebezpecnejsi (in Cz, includes negation and grade). Null if not requested/supported.

morphTag

morphTag: null | string

Morphological tag, e.g. AAMS1-...., VBD, ... Null if not requested/supported.

origCharSpan

origCharSpan: CharSpan

Character span within the original paragraph.

origText

origText: string

Text of this token in the original paragraph.

parent

parent: null | Token = null

Node that this node depends on. Null for the root or if no tree-structure is in the analysis.

pos

pos: null | UPos

Google universal tag. Null if not requested/supported.

sentence

sentence: Sentence

Sentence this node belongs to. Can be null only during construction.

subFnc

subFnc: null | string

Dependency sub-function. None if not requested/supported.

text

text: string

Text of this token, possibly after correction.

Static ReadonlyFEAT_LEMMA_INFO

FEAT_LEMMA_INFO: "lemmaInfo"

Lemma info features, a list of strings.

Static ReadonlyFEAT_NEGATED

FEAT_NEGATED: "negated"

Key presence signifies it is a negated word, value = true.

Static ReadonlyFEAT_UNKNOWN

FEAT_UNKNOWN: "unknown"

Key presence signifies it is an unknown word, value = true.

Accessors

fullFnc

  • get fullFnc(): null | string

  • Full dependency function in the format {fnc}:{subFnc} if the sub-function is present. Otherwise it's the same as fnc.

    Returns null | string

Methods

coverage

  • coverage(reflexive?: boolean, ordered?: boolean): Token[]

  • All nodes dominated by this node.

    Parameters

    • reflexive: boolean = true

      Whether this node itself is included.

    • ordered: boolean = true

      Whether the result should be ordered by word order.

    Returns Token[]

depth

  • depth(): number

  • Depth of this node in the dependency tree.

    Returns number

    The distance, i.e. number of dependency edges, from the root of the sentence.

filteredInOrder

  • filteredInOrder(
        includeFilteredRoot?: boolean,
        skipPredicate: (n: Token) => boolean,
    ): Iterable<Token>

  • In-order iterator over the subtree of this node, optionally skipping some subtrees.

    Parameters

    • includeFilteredRoot: boolean = true

      If true, the nodes on which [skipPredicate] returns true are included in the result.

    • skipPredicate: (n: Token) => boolean

      When this predicate is true on any node, the node's subtree is not traversed (skipping only their subtrees).

    Returns Iterable<Token>

filteredPreOrder

  • filteredPreOrder(
        includeFilteredRoot?: boolean,
        skipPredicate: (n: Token) => boolean,
    ): Iterable<Token>

  • Pre-order iterator over the subtree of this node, optionally skipping some subtrees.

    Parameters

    • includeFilteredRoot: boolean = true

      If true, the nodes on which [skipPredicate] returns true are included in the result (skipping only their subtrees).

    • skipPredicate: (n: Token) => boolean

      When this predicate is true on any node, the node's subtree is not traversed.

    Returns Iterable<Token>

inOrder

isContinuous

isLeaf

  • isLeaf(): boolean

  • Check whether this node is a leaf (i.e has no dependents).

    Returns boolean

isNegated

  • isNegated(): boolean

  • True iff the token form contains a negation prefix.

    Returns boolean

isRoot

isUnknown

  • isUnknown(): boolean

  • True iff the token is unknown to the lemmatizer. The lemma provided is the same as the token itself.

    Returns boolean

leftChildren

next

offsetToken

  • offsetToken(offset: number): null | Token

  • Token following of preceding this token within the sentence.

    Parameters

    • offset: number

      Relative offset. The following tokens have a positive offse, preceding a negative one. The ext token has offset = 1.

    Returns null | Token

    The token at the relative offset or null if the offset is invalid.

preOrder

previous

rigthChildren

toIndentTreeString

  • toIndentTreeString(
        printToken: (token: Token) => string,
        indentStr?: string,
        depth?: number,
    ): string

  • Indented string representation of the subtree of this node.

    Parameters

    • printToken: (token: Token) => string

      A function printing individual nodes.

    • indentStr: string = " "

      A string used to indent each level from the previous one.

    • depth: number = 0

      Indentation level to start with.

    Returns string

toSimpleIndentString

toSimpleString

toSimpleTreeString

toString

  • toString(): string

  • Returns a string representation of an object.

    Returns string

toStringWith

  • toStringWith(text: boolean, pos: boolean, fnc: boolean): string

  • Converts the token to a non-recursive string: index + [text] + [pos] + [fnc].

    Parameters

    • text: boolean
    • pos: boolean
    • fnc: boolean

    Returns string

toTreeString

  • toTreeString(printToken: (token: Token) => string): string

  • Parenthesised representation of the subtree of this node.

    Parameters

    • printToken: (token: Token) => string

      Function printing individual nodes.

    Returns string

    A string representation of the tree rooted in this node.

Staticof

  • of(
        id: string,
        idx: number,
        text: string,
        charSpan: CharSpan,
        origText?: null | string,
        origCharSpan?: null | CharSpan,
        deepLemma?: null | string,
        lemma?: null | string,
        pos?: null | UPos,
        morphTag?: null | string,
        feats?: null | Map<string, string>,
        fnc?: null | UDep,
        subFnc?: null | string,
    ): Token

  • Token factory method, public constructor.

    Parameters

    • id: string
    • idx: number
    • text: string
    • charSpan: CharSpan
    • origText: null | string = null
    • origCharSpan: null | CharSpan = null
    • deepLemma: null | string = null
    • lemma: null | string = null
    • pos: null | UPos = null
    • morphTag: null | string = null
    • feats: null | Map<string, string> = null
    • fnc: null | UDep = null
    • subFnc: null | string = null

    Returns Token

Settings

Member Visibility

On This Page

Properties
Accessors
Methods