diffutron
/

DiffutronLM-0.3B-Instruct

@@ -13,197 +13,82 @@ base_model:
 pipeline_tag: text-generation
 ---
-# Model Card for Model ID
-<!-- Provide a quick summary of what the model is/does. -->
-## Model Details
-### Model Description
-<!-- Provide a longer summary of what this model is. -->
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-### Model Sources [optional]
-<!-- Provide the basic links for the model. -->
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-## Uses
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
-### Direct Use
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-[More Information Needed]
-### Downstream Use [optional]
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-[More Information Needed]
-### Out-of-Scope Use
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-[More Information Needed]
-## Bias, Risks, and Limitations
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-[More Information Needed]
-### Recommendations
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
-## How to Get Started with the Model
-Use the code below to get started with the model.
-[More Information Needed]
-## Training Details
-### Training Data
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-[More Information Needed]
-### Training Procedure
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-#### Preprocessing [optional]
-[More Information Needed]
-#### Training Hyperparameters
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-#### Speeds, Sizes, Times [optional]
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
-[More Information Needed]
-## Evaluation
-<!-- This section describes the evaluation protocols and provides the results. -->
-### Testing Data, Factors & Metrics
-#### Testing Data
-<!-- This should link to a Dataset Card if possible. -->
-[More Information Needed]
-#### Factors
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-[More Information Needed]
-#### Metrics
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-[More Information Needed]
-### Results
-[More Information Needed]
-#### Summary
-## Model Examination [optional]
-<!-- Relevant interpretability work for the model goes here -->
-[More Information Needed]
-## Environmental Impact
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-## Technical Specifications [optional]
-### Model Architecture and Objective
-[More Information Needed]
-### Compute Infrastructure
-[More Information Needed]
-#### Hardware
-[More Information Needed]
-#### Software
-[More Information Needed]
-## Citation [optional]
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-**BibTeX:**
-[More Information Needed]
-**APA:**
-[More Information Needed]
-## Glossary [optional]
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-[More Information Needed]
-## More Information [optional]
-[More Information Needed]
-## Model Card Authors [optional]
-[More Information Needed]
-## Model Card Contact
-[More Information Needed]

 pipeline_tag: text-generation
 ---
+# DiffutronLM-0.3B-Instruct
+**Diffutron** is a parameter-efficient, Masked Diffusion Language Model (MDLM) specifically designed for the Turkish language. Unlike standard autoregressive models that generate text one token at a time, Diffutron generates text by iteratively refining sequences in parallel, allowing for simultaneous consideration of the entire sentence context.
+Despite its compact size of 307 million parameters, `DiffutronLM-0.3B-Instruct` achieves highly competitive performance against much larger, multi-billion-parameter autoregressive baselines on Turkish NLP benchmarks.
+## 📌 Model Details
+* **Model Type:** Masked Diffusion Language Model (MDLM)
+* **Base Architecture:** `jhu-clsp/mmBERT-base` (Multilingual Encoder)
+* **Language:** Turkish
+* **Parameter Count:** 307M (0.3B)
+* **Context Length:** 256 tokens (Instruct), 512 tokens (Base)
+* **Training Libraries:** `dllm`, PyTorch
+## 🚀 Architecture & Approach
+Diffutron departs from traditional next-token prediction. It treats text generation as a discrete diffusion process:
+1. **Forward Process:** Clean text is gradually corrupted into a sequence of `<mask>` tokens.
+2. **Reverse Process:** The model learns to denoise the sequence globally, attending to visible context bi-directionally to predict the original tokens.
+This non-autoregressive paradigm compresses linguistic knowledge efficiently, allowing this 0.3B model to punch significantly above its weight class.
+## 📚 Training Pipeline
+The model was developed through a resource-efficient, multi-stage training pipeline:
+### 1. Continual Pre-training (CPT)
+To adapt the multilingual backbone to Turkish without catastrophic forgetting, we employed a high-rank LoRA strategy (r=256, α=256) targeting all linear modules (Attention and MLP).
+* **Data:** ~2 million sequences sourced from Havadis (news), Temiz-OSCAR (web), and Turkish Wikipedia.
+* **Result:** Perplexity on the Bilkent Turkish Writings Dataset dropped significantly from 3.42 (base) to 2.75.
+### 2. Progressive Instruction-Tuning (SFT)
+To unlock generative instruction-following capabilities, we utilized a two-stage supervised fine-tuning approach:
+* **Stage 1 (General Adaptation):** Trained on `metunlp/LlamaTurk-Instruction-Set` for 20 epochs to establish fundamental instruction-following behaviors.
+* **Stage 2 (Complex Specialization):** Trained on the nuanced `turkish-nlp-suite/InstrucTurca` dataset for 8 epochs with an increased batch size, enhancing the model's ability to handle intricate, domain-specific Turkish commands.
+## 📊 Evaluation Results
+The model was evaluated on a representative subset of the **CETVEL Benchmark Suite**. DiffutronLM-0.3B (2nd Stage) demonstrates remarkable parameter efficiency, outperforming models up to 7x its size (e.g., Kumru-2B and TURNA-1.1B) on average scores.
+| Benchmark | Diffutron-1st (0.3B) | Diffutron-2nd (0.3B) | TURNA (1.1B) | Kumru (2B) | Kanarya (2B) | Llama-3.2 (3B) | Trendyol (7B) | Aya-101 (13B) |
+| :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |
+| **Belebele_TR** | 22.22 | 27.00 | 22.56 | 29.00 | 28.11 | **55.78** | 36.22 | 22.89 |
+| **EXAMS_TR** | 25.95 | 27.74 | 23.66 | **30.03** | **30.03** | 26.21 | 28.50 | 22.90 |
+| **IronyTR** | 50.67 | **52.00** | 48.33 | 51.00 | 50.00 | 50.17 | 50.00 | **52.17** |
+| **News_Cat** | 23.20 | 32.40 | 32.80 | 26.40 | 66.80 | 64.00 | **81.20** | 20.00 |
+| **MNLI_TR** | 33.29 | 32.81 | 34.94 | **36.42** | 33.40 | 34.76 | 35.19 | 27.90 |
+| **STS_TR** | 17.77 | **18.78** | 14.21 | 11.75 | 12.91 | 12.91 | 15.52 | 16.97 |
+| **XCOPA_TR** | 53.80 | 52.00 | 55.80 | 54.00 | **64.20** | 54.60 | 61.00 | 59.60 |
+| **Average** | 32.41 | **34.68** | 33.19 | 34.09 | 40.78 | 42.63 | **43.95** | 31.78 |
+## 💻 Usage
+Because Diffutron is a Masked Diffusion Language Model, it requires inference strategies distinct from standard causal generation. We recommend using the `dllm` library or custom generation loops tailored for discrete diffusion.
+### Generation Parameters Used in Paper:
+* **Longer Context:** Steps: 128, Temp: 0.1, Block Length: 32, Repetition Penalty: 1.2
+* **Shorter Context:** Steps: 64, Remask: `low_conf`, Stochastic: `False`, CFG: 0.0
+## ⚠️ Limitations
+* **Multilingual Backbone:** Built upon a multilingual encoder rather than a native Turkish foundation model.
+* **Context Window:** Restricted to a 256-token context window for generation, limiting its use in long-form summarization or document-level generation.
+* **Data Nuances:** Instruction datasets rely heavily on translations or synthetic data, which may occasionally miss subtle cultural contexts.
+## 📝 Citation
+If you use Diffutron in your research, please cite our preprint:
+```bibtex
+@misc{diffutron2026,
+  author = {Kocabay, Şuayp Talha and Akkuş, Talha Rüzgar},
+  title = {Diffutron: A Masked Diffusion Language Model for Turkish Language},
+  year = {2026},
+  publisher = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/collections/diffutron/diffutronlm}}
+}
+```