README.md

---
license: mit
---

# 🛠️ Gemma 4 31B SwiGLU Fusion Toolkit

This toolkit provides a robust pipeline for generating and injecting custom, functionally correct MLP weights into a Gemma 4 31B safetensors model. It is specifically designed to handle Gemma 4's unique architectural features—namely its interleaved sliding-window attention (SWA) and global full-context layers—while preserving model stability through advanced alignment and fusion techniques.

## ✨ Features

*   **Targeted Architecture:** Explicitly accounts for Gemma 4's interleaved attention structure (5 SWA + 1 Global per period).
*   **Double-Wide MLP Support:** Automatically detects and generates intermediate sizes for global full-context layers.
*   **Sign-Symmetric Alignment:** Generates weights using paired frame vectors ($+gc+gc$ and $-gc-gc$) to ensure the net activation remains zero-centered, preventing distribution shifts.
*   **Shape-Contoured Fusion (SCF):** Instead of naive addition, the fusion process smoothly merges synthetic weights into the model's existing learned contours, minimizing destabilization.
*   **Controlled Adjustment:** Implements dynamic scaling ($\alpha$) and clamping ($\gamma$-cap) to control the magnitude of weight updates.
*   **Selective Fusion:** Allows users to target specific layers for modification.

## ⚙️ Pipeline Overview

The toolkit operates in two stages, assuming you have generated the source Neuron and derivitive Neurons which are re-usable:

### 1. The Generator (`weights.py`)

This script creates synthetic, functionally correct MLP projections (`gate_proj`, `up_proj`, `down_proj`) based on reference source neurons. It maps piece-wise linear behaviors into the non-linear SwiGLU block without introducing mean-shift destabilization.

*   **Key Function:** Generates balanced, sign-symmetric delta weights.
*   **Output:** Saves the generated delta weights to an output directory.

### 2. The Fuser (`applyweights.py`)

This script applies the safetensor delta files generated by `weights.py` to your existing Gemma 4 31B model using Shape-Contoured Fusion (SCF).

*   **`down_proj` (Contoured Multiplicative Delta):** The update is scaled by the existing weight profile and variance-normalized to ensure smooth integration.
    $$\text{W}_{\text{down}} = \text{W}_{\text{down}} + (\alpha_{\text{dynamic}} \cdot \Delta_{\text{down}} \cdot \text{W}_{\text{down}})$$
*   **`gate_proj` (Multiplicative Gamma Scaling):** Applies a clamped fractional adjustment to prevent unbounded activation spikes.
    $$\text{W}_{\text{gate}} = \text{W}_{\text{gate}} \cdot (1 + \text{clamp}(\Delta_{\text{gate}}, -\gamma, \gamma))$$
*   **`up_proj` (Untouched):** The linear value path is intentionally skipped during fusion.

## 🚀 Usage

### Stage 1: Generating Delta Weights

Use `weights.py` to generate the synthetic delta weights.

```bash
python weights.py \
    --base-model /path/to/original_gemma_model \
    --output-dir generated_weights \
    --seed 42
```

### Stage 2: Applying Weights (Fusion)

Use `applyweights.py` to merge the generated weights into your target model.

```bash
python applyweights.py \
    --model /path/to/original_gemma_model \
    --weights generated_weights \
    --output /path/to/fused_gemma_model \
    --alpha 0.02 \
    --gamma-cap 0.05
```

## 💡 Helpful Flags

| Flag | Description | Default |
| :--- | :--- | :--- |
| `--dry-run` | Simulates the fusion process, identifying missing keys, calculating partial coverage matrices, and verifying sharded indexing without writing files. | N/A |
| `--layers` | A space-separated list of specific layer indices to fuse (e.g., `--layers 5 11 17`), skipping the rest. | None |
| `--alpha` | Controls the variance scale multiplier for the `down_proj` update. | `0.02` |
| `--gamma-cap` | Sets the maximum fractional adjustment allowed for the `gate_proj`. | `0.05` |

# Note: 

The default values for Alpha and Gamma are extremely conservative, they will not influence model behavior much at the defaults; it can and should, for pronounced change closer to Claude be pushed to around Alpha ~32 and Gamma-cap ~25 without breaking.