glrmask

GLRMask is a grammar-constrained decoding library for high-throughput LLM serving. It is optimized for fast mask computation with extremely low tail latency, even for complex grammars.

GLRMask supports general context-free grammars, including ambiguous ones. It accepts JSON schemas, GLRM, Lark, and EBNF.

Website ยท Getting started

GLRMask has three core types:

  • Grammar: the language you want to generate.
  • Constraint: a grammar compiled against a specific vocabulary.
  • ConstraintState: the mutable state for a single generated sequence.

Compile a Grammar into a Constraint, then call constraint.start(). In the decoding loop, compute the next-token mask, sample a token the mask allows, and commit that token to the state.

Quick start

import glrmask

vocab = glrmask.Vocab.from_id_to_bytes({
    0: b"hello",
    1: b" ",
    2: b"world",
})
grammar = glrmask.Grammar.from_ebnf(
    'start ::= "hello" " " "world"'
)
constraint = grammar.compile(vocab)
state = constraint.start()
assert state.mask()[0] & 1
state.commit_token(0)
state.commit_token(1)
state.commit_token(2)
assert state.is_accepting()

Vocab records the model token IDs and their bytes. Python masks are packed 32-bit words; bit token_id % 32 in word token_id // 32 indicates whether that token is allowed.

Use Grammar.compile_unlinked and UnlinkedConstraint when a compiled parent will be reused with different compiled children or exact-token bindings. The class and method reference below documents these operations.

class BoundaryTriggerDetail:

Optional boundary-query metadata, built before compile or link returns.

class Constraint:

Compiled grammar constraint. Immutable, thread-safe.

def save(self, /, *, external_vocab=False):

Serialize the compiled body and final termination policy as bytes.

external_vocab=False embeds the vocabulary. With external_vocab=True, loading requires the exact same vocabulary mapping, not just its size.

def load(data, vocab=None):
def start(self, /):
def mask_len(self, /):
class ConstraintState:

Mutable per-sequence parse state.

def clone(self, /):

Copy this sequence's exact parser state without sharing mutable state.

def mask(self, /, size=None):
def fill_mask(self, /, bitmask):
def commit_token(self, /, token_id):
def commit_bytes(self, /, data):
def is_rejected(self, /):
def forced(self, /):
def is_accepting(self, /):
class ExactToken:

One exact model token, retaining its complete vocabulary identity.

id
class ExactTokens:

A nonempty set of exact model tokens, retaining its vocabulary identity.

ids
class Grammar:

Immutable grammar description with source-grammar or exact-token bindings.

bind returns another description. compile produces a complete runnable Constraint; compile_unlinked intentionally preserves unresolved slots.

def from_ebnf(source):

Create a grammar from EBNF source text.

def from_lark(source):

Create a grammar from Lark source text.

def from_glrm(source):

Create a grammar from native GLRM source text.

def from_json_schema(schema):

Create a grammar from JSON Schema text or a JSON-serializable object.

def bind(self, /, name, value):

Return a new description with one declared slot bound. No compilation occurs.

def compile( self, /, vocab, *, end_tokens=None, optimization=None, boundary_trigger=None):

Compile a closed, immediately runnable constraint for this vocabulary.

end_tokens are reserved generation controls, allowed only at acceptance. Their policy is not inherited when the result is later embedded as a child.

def compile_unlinked(self, /, vocab, *, boundary_trigger=None):

Compile reusable machinery while preserving unresolved grammar/token slots.

class Optimization:

Preferred compilation/runtime trade-off; accepted-language semantics are unchanged.

FAST_RUNTIME = Optimization.FAST_RUNTIME
class UnlinkedConstraint:

Immutable vocabulary-specific compiled constraint in pre-link form.

def bind(self, /, name, value):

Bind a compiled Constraint child or exact-token value, returning a new value.

def save(self, /):

Serialize compiled machinery and its open-slot manifest as bytes.

def load(data, vocab=None):

Load an unlinked constraint, retaining open slots and exact vocabulary identity.

class Vocab:
def token(self, /, id):

Resolve one exact token, preserving this complete vocabulary identity.

def tokens(self, /, ids):

Resolve a nonempty set of distinct exact token IDs.

def from_dict(token_to_id):
def from_id_to_bytes(id_to_bytes):
def from_llama_cpp(llm):

Build the byte vocabulary used by a llama-cpp-python Llama model.

EOG, control, unused, and empty-piece tokens are omitted from the byte vocabulary. EOG IDs remain available through llama_cpp_end_token_ids for the decoder's stopping policy.

llama_cpp_end_token_ids