ConvexTok turns tokenizer construction into a shortest-path and sparse linear-programming problem over a byte-boundary DAG. ToricGT uses that DAG as real model input: byte boundaries are nodes, candidate substrings are edges, LP relaxation scores are edge features, and the final segmentation is the selected path used for BPB scoring.
The key point is not merely “a different tokenizer.” The tokenization computation is itself a graph-structured dynamic program, so it matches TokenGT-style graphification instead of being hidden preprocessing.
Tokenization DAG
Click an arc or boundary vertex.
Edge metadata will appear here without loading any external plotting runtime.
Curved arcs span byte intervals. Cyan bottom arcs are fallback byte edges, violet arcs are priced candidate tokens, gold arcs are LP-relaxation support, and green arcs are the selected rounded token path. The model receives the same structure as TokenGT-style node/edge features; BPB is still scored on the flattened selected path.
Equations
D[j] = min(i,j,t)∈E D[i] + wt
min Σe cexe subject to flow conservation and xe ≤ ytoken(e), Σt yt ≤ B.
The min-plus recurrence is tropical dynamic programming. The LP lower bound gives a tokenizer-regret metric: path length minus relaxed optimum. ToricGT can log this gap to decide whether BPB is tokenizer-bound or model-bound.
Boundary node
Byte offset in the source string; causal topological order is left to right.
Candidate edge
A token proposal from offset i to j, carrying byte length, token rank, price, LP score, and selected-path flags.
Tropical path
The rounded segmentation is a min-plus path through the DAG; active edges form a tropical curve inside the tokenizer graph.
Candidate Edge Metadata
span
token
kind
cost
LP
selected
0→1
t
free byte fallback
1.000
0.00
no
1→2
o
free byte fallback
1.000
0.00
no
2→3
k
free byte fallback
1.000
0.00
no
3→4
e
free byte fallback
1.000
0.00
no
4→5
n
free byte fallback
1.000
0.00
no
5→6
free byte fallback
1.000
0.00
no
6→7
g
free byte fallback
1.000
0.00
no
7→8
r
free byte fallback
1.000
0.00
no
8→9
a
free byte fallback
1.000
0.00
no
9→10
p
free byte fallback
1.000
0.00
no
10→11
h
free byte fallback
1.000
0.00
no
0→5
token
selected rounded token
0.270
0.82
yes
0→6
token
priced vocabulary token
0.350
0.48
no
6→11
graph
selected rounded token
0.210
0.77
yes
6→10
grap
priced vocabulary token
0.290
0.32
no
0→11
token graph
LP-relaxation long-span candidate
0.490
0.41
no
1. BPB Use
FineWeb is graphified, but the OAI scoring path still flattens the selected token path. This keeps the byte objective primary while letting the model learn node/edge structure around the tokenizer.
2. Tropical Use
The dynamic program D[j]=min(D[i]+w) is min-plus computation. Active edges, margins, and path regret become tropical diagnostics instead of opaque tokenizer side effects.
3. Toric Use
Candidate-token exponents and LP scores define active faces of a tokenization polytope. Those faces are embedded into toric charts for fan, one-dimensional-cone, divisor, and sheaf-style audits.
4. Graph Use
Boundary nodes, candidate-token edges, endpoint offsets, edge ranks, selected-path flags, and LP scores are TokenGT-style features. The graph is causal left-to-right for FineWeb.
5. Regret Use
The LP lower bound gives a tokenizer-regret metric: if rounded path length is close to the LP optimum, BPB problems are more likely model-bound; otherwise tokenizer vocabulary or rounding is suspect.
6. OOD Use
Because substring choices are graph paths, the model sees reusable local graph grammar rather than isolated token IDs. That is the bridge to graph-structured biological and 3D tokenizers later.