Record Class SubwordPiece

java.lang.Object
java.lang.Record
opennlp.tools.tokenize.SubwordPiece
Record Components:
piece - The piece in the vocabulary's normalized form; must not be null or empty.
id - The non-negative vocabulary id of the piece.
start - The inclusive start offset in the original text.
end - The exclusive end offset in the original text; at least start.
public record SubwordPiece(String piece, int id, int start, int end) extends Record
One subword unit produced by a SubwordTokenizer, including the model representation and source range.

The piece string is in the tokenizer's normalized form and need not equal the input. start and end are UTF-16 offsets into the original text, so the surface associated with this piece is text.subSequence(start, end). A span can include adjacent source characters when normalization reorders characters. Pieces without source text, such as control symbols, report an empty span with start == end.

The piece, id and span triple follows the representation established by SentencePiece (Kudo and Richardson, 2018), whose SentencePieceText.SentencePiece message carries piece, id and a [begin, end) range into the input, and gives control symbols an empty surface with begin == end. The surface field of that message has no counterpart here because it is text.subSequence(start, end). The offsets are UTF-16 indices rather than the UTF-8 byte offsets used there.

Since:
3.0.0
See Also:

  • Constructor Summary

    Constructors
    Constructor
    Description
    SubwordPiece(String piece, int id, int start, int end)
    Instantiates a SubwordPiece.

  • Method Summary

    Modifier and Type
    Method
    Description
    int
    end()
    Returns the value of the end record component.
    final boolean
    equals(Object o)
    Indicates whether some other object is "equal to" this one.
    final int
    hashCode()
    Returns a hash code value for this object.
    int
    id()
    Returns the value of the id record component.
    String
    piece()
    Returns the value of the piece record component.
    int
    start()
    Returns the value of the start record component.
    final String
    toString()
    Returns a string representation of this record class.

    Methods inherited from class Object

    getClass, notify, notifyAll, wait, wait, wait

  • Constructor Details

    • SubwordPiece

      public SubwordPiece(String piece, int id, int start, int end)
      Instantiates a SubwordPiece.
      Throws:
      IllegalArgumentException - Thrown if piece is null or empty, id is negative, or the span is negative or inverted.

  • Method Details

    • toString

      public final String toString()
      Returns a string representation of this record class. The representation contains the name of the class, followed by the name and value of each of the record components.
      Specified by:
      toString in class Record
      Returns:
      a string representation of this object

    • hashCode

      public final int hashCode()
      Returns a hash code value for this object. The value is derived from the hash code of each of the record components.
      Specified by:
      hashCode in class Record
      Returns:
      a hash code value for this object

    • equals

      public final boolean equals(Object o)
      Indicates whether some other object is "equal to" this one. The objects are equal if the other object is of the same class and if all the record components are equal. Reference components are compared with Objects::equals(Object,Object); primitive components are compared with the compare method from their corresponding wrapper classes.
      Specified by:
      equals in class Record
      Parameters:
      o - the object with which to compare
      Returns:
      true if this object is the same as the o argument; false otherwise.

    • piece

      public String piece()
      Returns the value of the piece record component.
      Returns:
      the value of the piece record component

    • id

      public int id()
      Returns the value of the id record component.
      Returns:
      the value of the id record component

    • start

      public int start()
      Returns the value of the start record component.
      Returns:
      the value of the start record component

    • end

      public int end()
      Returns the value of the end record component.
      Returns:
      the value of the end record component