TL;DR: Gemma 4 E2B has ~50% PLE redundancy. You can surgically disable up to 46% of PLE layers with minimal quality loss. At 50%, the model visibly degrades. At 54%, it produces zero output.
Gemma 4 E2B uses Per-Layer Embeddings — each of its 28 decoder layers has its own embedding table. At every layer, token embeddings are looked up, gated by the current hidden state, and added to the residual stream. This is why the "effective" parameter count (2B) is much smaller than the total (~5B). PLE accounts for roughly 60% of the model's total parameters.
This study systematically disables PLE at specific layers and layer combinations to map the structural dependency of the model on this mechanism.
| Component | Details |
|---|---|
| GPU | AMD Radeon 8060S (Strix Halo), 128GB unified memory, gfx1151 |
| Model | gemma-4-e2b-it-Q8_0.gguf (4.97 GB, 28 layers) |
| Quant Source | ggml-org/gemma-4-E2B-it-GGUF |
| llama.cpp | Build b8635 (commit 5803c8d11) |
| ROCm | 7.2.0 |
We patched llama.cpp's Gemma 4 ISWA model implementation (src/models/gemma4-iswa.cpp) with environment variable toggles to control PLE at runtime without recompiling:
GEMMA4_DISABLE_PLE=1 # Kill all PLE (100% removal)
GEMMA4_PLE_STRIDE=N # Apply PLE every Nth layer only
GEMMA4_PLE_HALF=first|last # First or last half only
GEMMA4_PLE_SKIP_LAYER=0,1,2,14,27 # Skip specific layers (comma-separated)The patch is available at patches/gemma4-ple-ablation.patch. Apply it to llama.cpp build b8635:
cd /path/to/llama.cpp
git apply /path/to/patches/gemma4-ple-ablation.patch
cmake --build build -jPrompt: "What is the capital of France? Answer in one sentence."
Layers 0–27 are divided into three regions: Begin (0–8), Middle (9–18), End (19–27).
| Config | Layers Skipped | # Skipped | % Removed | Correct Answer? | Prompt t/s | Gen t/s | Notes |
|---|---|---|---|---|---|---|---|
| 2× Begin | 0, 1 | 2 | 7% | ✅ Yes | 323.0 | 67.9 | Perfect |
| 2× Middle | 13, 14 | 2 | 7% | ✅ Yes | 353.8 | 67.4 | Perfect |
| 2× End | 26, 27 | 2 | 7% | ✅ Yes | 355.2 | 67.4 | Perfect |
| 3× Begin | 0, 1, 2 | 3 | 11% | ✅ Yes | 352.6 | 67.3 | Perfect |
| 3× Middle | 13, 14, 15 | 3 | 11% | ✅ Yes | 351.5 | 67.7 | Perfect |
| 3× End | 25, 26, 27 | 3 | 11% | ✅ Yes | 322.6 | 67.6 | Minor truncation in thinking |
| 1+1+1 cross-region | 0, 14, 27 | 3 | 11% | ✅ Yes | 354.9 | 67.7 | Perfect |
| 2+2+2 cross-region | 0, 1, 13, 14, 26, 27 | 6 | 21% | ✅ Yes | 357.2 | 67.9 | Perfect |
| 3+3+3 cross-region | 0, 1, 2, 13, 14, 15, 25, 26, 27 | 9 | 32% | ✅ Yes | 329.9 | 68.4 | Perfect |
Key Finding: Up to 32% of PLE layers can be removed from any region with zero quality degradation on factual recall.
Results from progressive escalation, testing factual recall at each level:
| Layers Skipped | % Removed | Pattern | Result | Status |
|---|---|---|---|---|
| 1 | 3.6% | Single layer | Perfect output | ✅ |
| 3 (1+1+1) | 11% | Cross-region | Perfect output | ✅ |
| 6 (2+2+2) | 21% | Cross-region | Perfect output | ✅ |
| 9 (3+3+3) | 32% | Cross-region | Perfect output | ✅ |
| 12 (4+4+4) | 43% | Cross-region | Perfect output | ✅ |
| 13 (5+4+4) | 46% | Cross-region | Subtle degradation | |
| 14 (5+4+5) | 50% | Cross-region | Visible degradation | |
| 15 (5+5+5) | 54% | Cross-region | Zero output | ❌ |
| 14 (stride=2) | 50% | Every other layer | Multilingual hallucination | ❌ |
| 28 (all) | 100% | Complete removal | Zero output | ❌ |
This is the critical comparison at the degradation cliff. Both configurations were tested with 10 diverse prompts across factual recall, math, translation, coding, and reasoning.
| # | Prompt | Result | Quality |
|---|---|---|---|
| 1 | What is the capital of France? | "The capital of France is Paris." | ✅ |
| 2 | List the first 5 prime numbers | Correct numbers (2,3,5,7,11) but wrong list numbering (1,2,4,6,11) | |
| 3 | Translate "Good morning" to ES/FR/DE | Buenos días, Bonjour, Guten Morgen — correct but verbose/repetitive | ✅ |
| 4 | What is 17 × 23? | Answers $117$ (wrong — correct is 391) |
❌ |
| 5 | Write a Python factorial function | Clean iterative implementation with docstring | ✅ |
| 6 | Who painted the Mona Lisa? | "Attributed to an artist known for that work" — won't say Da Vinci | |
| 7 | Chemical formula for water? | H₂O — correct with minor LaTeX verbosity | ✅ |
| 8 | Bash one-liner to count lines in a file | Discusses wc -l correctly in thinking, garbled output |
|
| 9 | Why is the sky blue? (2 sentences) | Correct Rayleigh scattering explanation, cut off before finishing | |
| 10 | Python palindrome checker | Starts correct implementation, cut off before completion |
Score at 46%: 4/10 ✅ perfect, 5/10
| # | Prompt | Result | Quality |
|---|---|---|---|
| 1 | What is the capital of France? | Thinking says "Italy" but answers "Paris" correctly | |
| 2 | List the first 5 prime numbers | Says "first 4" then adds 11 — confused structure | |
| 3 | Translate "Good morning" to ES/FR/DE | Garbled thinking ("Good\ up"), final answer: * **** (empty) |
❌ |
| 4 | What is 17 × 23? | Degenerate counting loop: "2+1=1, 12+1=13, 13+1=14..." | ❌ |
| 5 | Write a Python factorial function | Thinking is decent but gets cut off before producing code | |
| 6 | Who painted the Mona Lisa? | Hallucinates "La Jeune Jolie", "Marily/Marie" — can't identify painting | ❌ |
| 7 | Chemical formula for water? | Garbled LaTeX: $\$\$\$\text{H}\text{O$}$ |
❌ |
| 8 | Bash one-liner to count lines | Proposes y = | cat -n < file — nonsense |
❌ |
| 9 | Why is the sky blue? | Actually decent! Correct Rayleigh scattering explanation | |
| 10 | Python palindrome checker | Good thinking about algorithm but no code produced |
Score at 50%: 0/10 ✅ perfect, 4/10
| Prompt | 13 skipped (46%) | 14 skipped (50%) | Delta |
|---|---|---|---|
| Capital of France | ✅ | -1 | |
| First 5 primes | = | ||
| Translation (ES/FR/DE) | ✅ | ❌ | -2 |
| 17 × 23 | ❌ | ❌ | = |
| Python factorial | ✅ | -1 | |
| Mona Lisa painter | ❌ | -1 | |
| H₂O formula | ✅ | ❌ | -2 |
| Bash line count | ❌ | -1 | |
| Sky is blue | = | ||
| Palindrome checker | = |
- ✅ Factual recall works (Paris, translations, primes)
- ✅ Code generation still functional
⚠️ Math is broken (17×23 → 117 instead of 391)⚠️ Entity recall degrades (knows Mona Lisa exists, can't say "Da Vinci")- The model loses precision but retains structure
⚠️ Factual recall partially works but thinking is confused ("Italy" → "Paris")- ❌ Math completely broken (degenerate counting loops)
- ❌ Translation fails (garbled output)
- ❌ Entity recall broken (hallucinates painting names)
⚠️ Scientific reasoning surprisingly intact (Rayleigh scattering)- The model loses both precision and reliability but retains some reasoning structure
- ❌ Zero tokens produced — complete model collapse
- No single PLE layer is critical — removing any individual layer has zero measurable impact
- The failure is collective — it takes removing ~half the PLE layers to cause degradation
- Math breaks first — arithmetic is the first capability to degrade (at 46%)
- Entity recall degrades before general reasoning — can't name "Da Vinci" but can explain Rayleigh scattering
- The pattern suggests PLE works as a distributed ensemble — each layer contributes partial information, with enough redundancy to tolerate ~45% loss
- Stride-based removal is worse than block-based — removing every other layer (stride=2, 50%) causes hallucination, while block removal at the same percentage causes garbled output. This suggests adjacent PLE layers form functional groups.
100% PLE ──────────────── All capabilities intact
↓
~45% removal ─────────── Math/arithmetic fails first
↓
~46% removal ─────────── Entity recall degrades (proper nouns)
↓
~50% removal ─────────── Translation, code generation fail
↓
~50% (stride) ────────── Multilingual hallucination
↓
~54% removal ─────────── Complete output collapse (zero tokens)
| File | Description |
|---|---|
patches/gemma4-ple-ablation.patch |
The llama.cpp diff that adds env-var PLE controls |
batch_skip_test.sh |
Test script for regional block removal (Experiment 1) |
threshold_test.sh |
Test script for 13 vs 14 layer deep-dive (Experiment 3) |
run_ab_test.sh |
A/B test with PLE fully on vs fully off |
results.txt |
Raw output from Experiment 1 |
threshold_results.txt |
Raw output from Experiment 3 |
# 1. Clone and build llama.cpp with ROCm
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout b8635 # or later — patch applies to the gemma4-iswa model file
# 2. Apply the ablation patch
git apply /path/to/patches/gemma4-ple-ablation.patch
# 3. Build with ROCm (adjust for your GPU)
cmake -B build -DGGML_HIP=ON -DAMDGPU_TARGETS="gfx1151"
cmake --build build -j
# 4. Download the model
huggingface-cli download ggml-org/gemma-4-E2B-it-GGUF gemma-4-e2b-it-Q8_0.gguf
# 5. Run with PLE controls
# Normal (PLE on):
./build/bin/llama-cli -m gemma-4-e2b-it-Q8_0.gguf -ngl 99 -p "Hello"
# PLE disabled entirely:
GEMMA4_DISABLE_PLE=1 ./build/bin/llama-cli -m gemma-4-e2b-it-Q8_0.gguf -ngl 99 -p "Hello"
# Skip specific layers:
GEMMA4_PLE_SKIP_LAYER="0,1,2,13,14,15,25,26,27" ./build/bin/llama-cli -m gemma-4-e2b-it-Q8_0.gguf -ngl 99 -p "Hello"The ablation patch and test scripts are MIT licensed. The Gemma 4 model itself is subject to Google's Gemma Terms of Use.