SURFACE / ORDEREXACT FINITE CERTIFICATES

WORDS → FLOW → VOCABULARY → SHORTEST PATH

Choose a vocabulary. See the gap.

ConvexTok relaxes discrete token choices into fractional capacities. Here, five tiny CPU examples compare an exact rational LP lower bound, an exhaustively optimal vocabulary, and three rounding rules. Arrow words come from actual genus-two and genus-four infinite locally gentle algebras.

Loading exact finite evidence…

min Σe xe   subject to   Bx = d,   0 ≤ x,c ≤ 1,   xe ≤ ctoken(e),   Σt ct ≤ K.

Single symbols are always available and still cost one token per use. K counts additional token types. A token can occur repeatedly: each occurrence has its own capacity constraint. Edges never cross corpus-word boundaries.

Vertices are positions in one word, not nodes of the surface quiver. Cyan edges follow the optimal discrete segmentation; amber edges carry fractional LP flow. Hover or focus an edge for its flow and capacity. Arc height encodes span, not a metric.

Encode and recover every symbol

A fixed-vocabulary trap

With tokens {ab, bcd} and word abcd, longest-first greedy gives ab | c | d (3). The shortest-path recurrence gives a | bcd (2). This is distinct from BPE's choice of which tokens to create.

How the two bounds are checked

For min qx with Ax=b, Gx≤h and x≥0, the saved witnesses satisfy x≥0, Ax=b, Gx≤h, z≤0 and Aᵀy+Gᵀz≤q, with qx=by+hz. All upper bounds appear as rows of G. Python checks every entry using exact fractions after SciPy proposes a solution. Equality of feasible primal and dual values certifies this finite LP optimum. The integer optimum separately exhausts every size-K subset; adding tokens cannot increase its shortest-path cost.

All literal rational witnesses · Executable checks

What transfers to the surface-order project

For m=2g+1, the clean dessin on y²=xᵐ−1, β=xᵐ has σ=(0 … m−1)(m … 2m−1), α(h)=h+m mod 2m, permitted successor τ=σ⁻¹, and forbidden successor φ=ατ. The corpus follows permitted arrows ahaτh… in the infinite polynomial algebra Ag=kQ/(ahaφh). Products here use travel order: ab traverses a first, then b; left-operator matrix products reverse this order. Vocabulary tokens name finite subpaths; decoding recovers the same arrow sequence, endpoints and length.

The genus-two/four examples restrict candidate tokens to two or three arrows, so their lower bounds concern that candidate set. They compress a finite path corpus and preserve the algebra's relations. The completed order has rank 2m² over k[[u]]; this is unrelated to the number of tokens. Idempotents, arbitrary formal sums, path multiplication and module maps require their own representation. TokenGT's node/edge feature tokens have a different meaning; replacing them would require explicit incidence metadata, model changes and new evaluation.

Tempus et al., Tokenisation via Convex Relaxations, v1 supplies the formulation and rounding rules. The six-word vocabulary trap reproduces its small example; the gap and surface corpora, rational certificates and this interface are original adaptations. No language model is trained here; the paper's downstream benchmarks, GPU-scale optimization and its empirical near-optimality results are not reproduced. This tool establishes no arithmetic determinant bridge or GRH result.

Primary-source review and limits · Reproduction instructions · Mathematical exposition in the book