Neural Alignment

Prompt Browser

Note that both compressed models studied here, MKA and MKA with neural alignment, merge 11 layers, which corresponds to a compression ratio of 34.375 percent. Use the controls below to explore how the different models responded to each prompt.

Model: Original model MKA compressed model Neural alignment model Prompt:

Prompt

Ground truth

Model output

Accuracy vs. Compression Ratio (Neural Alignment Method)

The plot and table below compare our neural alignment merging strategy with the original MKA heuristic across different compression ratios. While MKA shows strong accuracy at low compression (approximately 3 percent), it suffers from a large performance collapse at 12.5 percent compression, dropping to 0.547. In contrast, our alignment method remains much more stable, achieving 0.633 at the same compression level, which is an improvement of 0.086 over MKA.

As compression increases, both methods converge to similar performance (approximately 0.63 at 40 percent), but neural alignment avoids the severe volatility exhibited by MKA. This indicates that correctly aligning neurons before merging leads to a more reliable and predictable degradation curve, especially in the moderate compression regime where real world deployments often operate.

Layers	Compression (%)	Ours (Accuracy)	MKA (Accuracy)
1	3.125	0.6478	0.6620
4	12.500	0.6334	0.5470
11	34.375	0.6124	0.6487
13	40.625	0.6392	0.6342
15	46.875	0.2631	0.3000