gm.testing.DummyTokenizer

class gemma.gm.testing.DummyTokenizer(

*,

path: str | os.PathLike = '/tmp/dummy_tokenizer.model',

custom_tokens: dict[int,

str]=<factory>,

)

Bases: gemma.gm.text._tokenizer.Tokenizer

Dummy tokenizer.

path: str | os.PathLike = '/tmp/dummy_tokenizer.model'

special_tokens

alias of gemma.gm.text._tokenizer._Gemma3SpecialTokens