A task-conditioned hypernetwork transformer prototype that generates bounded per-layer deltas and uses residual adapters as the actual controllable pathway, with early evidence on synthetic digit-reasoning tasks and an explicit proxy-failure diagnostic.

MetaModel

⚠️ Disclaimer — this is Phase 1 of a multi-phase project. The codebase represents the foundational scaffold only. All infrastructure (self-training, research agent, self-modification loop, self-critic) is built but untested under real autonomous conditions. The 340M model has not been trained. sum_mod3 is stuck at 32.7% on the 4.92M baseline. Do not take the current results as representative of the system's eventual capability — they are a working proof-of-concept for the architecture, not a finished system.

---

Theory

The problem: one model, many tasks

Standard models are trained for one purpose. To handle a new task, you either fine-tune (expensive, can forget old tasks) or maintain separate models (wasteful). The core idea here is different: make a single model that reconfigures its own weights based on what it's being asked to do.

What hypernetworks do

A hypernetwork is a neural network that outputs the weights of another network. Rather than having fixed parameters W, the model has a function H(t) that generates task-specific weights:

W_task = H(task_description)  
y = f(x, W_task)  

This is powerful because the same base network can solve fundamentally different tasks — not by learning a shared representation, but by generating different weight configurations per task. The hypernetwork learns how to reconfigure, not what to store.

Why weight deltas instead of full weights

Generating full weight matrices from scratch is expensive and hard to train. Instead, the hypernetwork generates deltas (ΔW) that modify a frozen base model:

W_final = W_base + α · ΔW  

The base model knows how to process sequences (via self-attention and FFN). The hypernetwork only needs to learn how to nudge those weights toward the right computation for each task. This keeps the model stable — the base stays general — while the deltas provide specialization.

The scalar α (learned per layer) controls how strongly to apply each delta. This is the Scope Predictor: it decides how task-specific a given layer needs to be.

The bottleneck trick (TaskAdapter)

Naive delta injection tends to either overfit (deltas too large) or have no effect (deltas too small). The TaskAdapter bridges this with a low-rank bottleneck:

ΔW = U · V^T   where U ∈ R^{H×r}, V ∈ R^{H×r}, r << H  

The hypernetwork generates U and V (rank-r matrices), and their product produces the delta. This constrains the delta to a low-dimensional subspace — enough to meaningfully modify behavior without destabilizing the base. It's the same principle as LoRA, but generated dynamically at inference time, conditioned on the task description rather than on a static adapter.

Closed-loop self-training (the AI-led research lab)

Training the hypernet doesn't end when loss converges. The system monitors for proxy shortcuts: the hypernet learns a correlation that works in training but doesn't implement the actual task. For example, sum_mod3 learned to correlate with digit magnitude (r=-0.43) instead of actual modulo-3 arithmetic.

When the SelfCritic detects a proxy, it:

1. Quarantines the proxy variant
2. Triggers the TaskGenerator to propose a harder variant of the same task
3. Routes the variant through MetaAdapterTrainer for fast fine-tuning
4. Merges the result back into the main training loop

The loop is autonomous: human sets strategy, the agent executes tactics, metrics drive routing decisions.

What this achieves vs. standard fine-tuning

---

Quick start

BASH
# Interactive REPL  
python3 main.py

# Single prediction  
python3 main.py --predict 42 fib_check

# Self-correction loop (retry until correct)  
python3 main.py --retry 1234 digit_sum

# Full benchmark on all 10 tasks  
python3 main.py --benchmark

# Train the model  
python3 main.py --train --epochs 50 --batch-size 64  

---

Architecture

Data flow

MERMAID
flowchart LR  
    A["task description\n\"sum of digits\""] --> B["HyperNetworkV3i\nLSTM"]  
    B --> C["weight deltas\nΔW per layer"]  
    C --> D["Base Transformer\nfrozen"]  
    A --> E["digit tokens\n1 2 3 4"]  
    E --> D  
    D --> F["PredictionHead\nper task"]  
    F --> G["class prediction"]  

Component interaction

MERMAID
flowchart TD  
    subgraph INPUT  
        T["task description"]:::box  
        D["digit input"]:::box  
        K["task ID (0-9)"]:::box  
    end

    subgraph HYPERNET["HyperNetworkV3i"]  
        ENC["TaskEncoder\nchar-level LSTM"]:::box  
        EMB["TaskEmbed\n128D → 256D"]:::box  
        LSTM["4-layer LSTM\nper-layer deltas"]:::box  
    end

    subgraph ADAPTER["TaskAdapter (rank r)"]  
        DOWN["H → r down-proj"]:::box  
        MID["SiLU + Dropout"]:::box  
        UP["r → H up-proj"]:::box  
        G["gate g ∈ [0,1]"]:::box  
    end

    subgraph BASE["Base Transformer"]  
        L1["LayerNorm + RoPE"]:::box  
        ATT["Multi-head\nSelf-Attention"]:::box  
        FFN["FFN"]:::box  
    end

    T --> ENC --> EMB --> LSTM  
    LSTM -->|ΔW_attn_q| ATT  
    LSTM -->|ΔW_attn_k| ATT  
    LSTM -->|ΔW_ffn| FFN  
    LSTM -->|g| G  
    K --> EMB  
    D --> L1

    ATT -. "delta\ninjection" .-> ADAPTER  
    FFN -. "delta\ninjection" .-> ADAPTER  
    ADAPTER -->|g · ΔW| ATT  
    ADAPTER -->|g · ΔW| FFN

    style HYPERNET fill:#1a1a2e,stroke:#7b68ee,color:#fff  
    style ADAPTER fill:#1a2a1a,stroke:#3cb371,color:#fff  
    style BASE fill:#1a1a2e,stroke:#4a9eff,color:#fff  
    style INPUT fill:#2a1a2e,stroke:#ff7b54,color:#fff

    classDef box fill:#2a2a4a,stroke:#aaa,color:#fff,rx:4  

Self-training loop

MERMAID
flowchart LR  
    E["Training Epoch"] --> P["PerformanceMonitor"]  
    P --> DIFF["DifficultyEstimator"]  
    DIFF --> CURR["CurriculumManager"]

    subgraph LOOP["SelfCritic loop"]  
        SC["SelfCritic.judge()\nproxy detection"]:::critic  
        TG["TaskGenerator\nnew variant"]:::gen  
        MA["MetaAdapterTrainer\nfast fine-tune"]:::train  
    end

    CURR -. "difficulty\nthreshold" .-> SC  
    P -. "accuracy\nmetrics" .-> SC  
    SC -->|quarantine proxy| MA  
    SC -->|new harder variant| TG  
    TG -->|variant spec| MA  
    MA -->|merged weights| P

    style LOOP fill:#0a1a0a,stroke:#3cb371,color:#fff

    classDef critic fill:#3a1a0a,stroke:#ff7b54,color:#fff  
    classDef gen fill:#1a2a3a,stroke:#4a9eff,color:#fff  
    classDef train fill:#2a1a3a,stroke:#9b59b6,color:#fff  

Key numbers (small model, verified)

Full model (planned): ~340M — same arch at scale.

---

Training

Run the training loop

BASH
# On-the-fly generation (no pre-processed data needed)  
cd training  
python3 train_v6.py --epochs 50 --batch-size 64 --lr 3e-4

# With pre-generated Arrow data (10-50x faster loading)  
python3 train_v6.py --epochs 50 --batch-size 64 --arrow-dir ../data/arrow

# Generate the Arrow data first  
cd ..  
python3 vectorize_dataset.py --split all  

Generate dataset

BASH
python3 vectorize_dataset.py                    # both train + val  
python3 vectorize_dataset.py --split train      # train only  
python3 vectorize_dataset.py --dry-run          # show what would be generated  

Training configuration

Edit TrainConfig in training/train_v6.py (line ~580):

Monitor training

Training writes to experiments_v6/training_history.json each epoch:

BASH
python3 -c "  
import json  
with open('experiments_v6/training_history.json') as f:  
    h = json.load(f)  
e = h[-1]  
print(f'Epoch {e[\"epoch\"]}: val={e[\"val_acc\"]:.1%} loss={e[\"val_loss\"]:.3f}')  
print('Task accuracies:', json.dumps(e['task_acc'], indent=2))  
"

---

Results (small model, 4.92M params)

28 epochs on RTX 3050 (~1.4h wall time):

Overall: 57.2% → 89.7% val accuracy (+32.6pp over 28 epochs)

sum_mod3 is the key failure

The hypernet learned a proxy that correlates with digit_sum (r=-0.43) but doesn't implement actual modulo-3. Every variant detected as a proxy by SelfCritic is correctly flagged — the infrastructure works. The issue is that with the current 4.92M model, there's insufficient capacity to learn the actual modulo operation, and the proxy provides a local minimum.

Fix path: Scale to the 340M model, which has enough capacity to learn sum_mod3 properly. The proxy-detection infrastructure (self_training/self_critic.py) will catch it if it recurs.

---

Project structure

MERMAID
flowchart RL  
    subgraph ROOT["meta_transformer/"]  
        subgraph CORE["core"]  
            MAIN["main.py"]:::file  
            RA["research_agent.py"]:::file  
            VD["vectorize_dataset.py"]:::file  
            DL["data_loader_arrow.py"]:::file  
            INIT["__init__.py"]:::file  
        end

        subgraph MODELS["models/"]  
            MTV["meta_transformer_v3.py"]:::file  
        end

        subgraph TRAIN["training/"]  
            TV["train_v6.py"]:::file  
        end

        subgraph ST["self_training/"]  
            MST["meta_self_trainer.py"]:::critic  
            SC["self_critic.py"]:::critic  
            CM["curriculum_manager.py"]:::gen  
            TG["task_generator.py"]:::gen  
            DE["difficulty_estimator.py"]:::gen  
            PM["performance_monitor.py"]:::gen  
            MA["meta_adapter_trainer.py"]:::train  
            RS["research_state.py"]:::gen  
        end

        subgraph INF["inference/"]  
            SML["self_modification_loop_v3.py"]:::file  
        end

        subgraph DATA["data/arrow/"]  
            TR["train/"]:::folder  
            VL["val/"]:::folder  
            SI["zd21_SciInstruct/"]:::folder  
        end

        subgraph EXP["experiments_v6/"]  
            BM["best_model.pt"]:::bin  
            CK["checkpoint_latest.pt"]:::bin  
            TH["training_history.json"]:::file  
            TL["training_log.txt"]:::file  
        end  
    end

    MAIN --> MTV  
    MAIN --> TV  
    RA --> ST  
    TV --> DL  
    VD --> DL  
    ST --> MTV

    style CORE fill:#0f1a2e,stroke:#4a9eff,color:#fff  
    style MODELS fill:#1a0f2e,stroke:#9b59b6,color:#fff  
    style TRAIN fill:#0f1a0f,stroke:#3cb371,color:#fff  
    style ST fill:#1f1a0a,stroke:#f39c12,color:#fff  
    style INF fill:#1a1a1a,stroke:#aaa,color:#fff  
    style DATA fill:#0f0f1a,stroke:#aaa,color:#fff  
    style EXP fill:#0f0f0f,stroke:#aaa,color:#fff

    classDef file fill:#1a2a4a,stroke:#4a9eff,color:#fff,rx:3  
    classDef folder fill:#1a2a1a,stroke:#3cb371,color:#fff,rx:3  
    classDef bin fill:#2a1a1a,stroke:#e74c3c,color:#fff,rx:3  
    classDef critic fill:#3a1a0a,stroke:#ff7b54,color:#fff,rx:3  
    classDef gen fill:#0a1a2a,stroke:#4a9eff,color:#fff,rx:3  
    classDef train fill:#1a0a1a,stroke:#9b59b6,color:#fff,rx:3  

Legend: self_training/ modules are colored by role — 🟠 critic, 🔵 generator, 🟣 trainer, 🟢 core.

---

Self-training loop

The architecture supports a closed-loop self-improvement cycle:

MERMAID
flowchart TD  
    subgraph TRAIN["Training Loop"]  
        EPO["Training Epoch"] --> PM["PerformanceMonitor"]  
        PM --> DE["DifficultyEstimator"]  
        DE --> CM["CurriculumManager"]  
        CM --> SC["SelfCritic.judge()"]  
        PM -. "accuracy\nmetrics" .-> SC  
    end

    subgraph CRITIC["Proxy Detection"]  
        SC -. "quarantine\nproxy" .-> Q["Quarantined"]  
        SC -. "new harder\nvariant" .-> TG["TaskGenerator"]  
        SC -. "verdict" .-> MS["MetaSelfTrainer"]  
    end

    subgraph ADAPTER["Adaptation"]  
        TG --> MA["MetaAdapterTrainer\nfast fine-tune"]  
        MA --> MW["Merged\nWeights"]  
        MW --> PM  
    end

    MS -->|"continue / stop\n/ quarantine"| EPO

    style TRAIN fill:#0d1f0d,stroke:#3cb371,color:#fff  
    style CRITIC fill:#1f0d0d,stroke:#ff7b54,color:#fff  
    style ADAPTER fill:#0d0d1f,stroke:#4a9eff,color:#fff  

Run the self-training harness:

BASH
python3 research_agent.py --mode agent  

Or drive it programmatically via self_training/meta_self_trainer.py.

---

Related work

This project builds on several lines of prior research:

---

What's next

• Train the 340M model — the architecture is ready, just needs GPU time (~6-12h on RTX 3050)
• Fix sum_mod3 at scale — 340M should have enough capacity to learn modulo-3 without proxy
• Run full self-training loop — current self-training modules are built but haven't been exercised on real training data
• Verify proxy detection — SelfCritic infrastructure exists but needs real difficulty-scaled variants to validate
• GSM8K evaluation — datasets/gsm8k/ data exists; evaluate transfer learning

---

Background

The core hypothesis: a hypernetwork that reads task descriptions and generates weight deltas can make a single model adapt to multiple tasks without fine-tuning. The architecture went through several iterations:

• v1/v2: Frozen base + hypernet → causality-inert deltas (deltas too small to change logits meaningfully)
• v3/v4: Joint base+hypernet training → marginal improvement
• v5: TaskAdapter bottleneck injection → significant improvement
• v6 (current): HyperNetworkV3i + TaskAdapter + PredictionHeads → 89.7% val accuracy

The stuck sum_mod3 task at the 4.92M scale is the primary remaining challenge. The self-training infrastructure was built specifically to detect and route around such proxy shortcuts.

Source: https://github.com/IlumCI/Metamodel