| --- |
| license: apache-2.0 |
| language: |
| - pt |
| pipeline_tag: text-generation |
| parameters: 10k |
| --- |
| |
| # MiniText-v1.0 |
|
|
| ## Model Summary |
|
|
| MiniText-v1.0 is a tiny **character-level language model** trained from scratch |
| to learn basic Portuguese text patterns. |
|
|
| The goal of this project is to explore the **minimum viable neural architecture** |
| capable of producing structured natural language, without pretraining, |
| instruction tuning, or external corpora. |
|
|
| This model is intended for **research, education, and experimentation**. |
|
|
| --- |
|
|
| ## Model Details |
|
|
| - **Architecture:** Custom MiniText (character-level) |
| - **Training Objective:** Next-character prediction |
| - **Vocabulary:** Byte-level (0–255) |
| - **Language:** Portuguese (basic) |
| - **Initialization:** Random (no pretrained weights) |
| - **Training:** Single-stream autoregressive training |
| - **Parameters:** ~10K |
|
|
| This is a **base model**, not a chat model. |
|
|
| --- |
|
|
| ## Training Data |
|
|
| The model was trained on a **synthetic Portuguese dataset** designed to emphasize: |
|
|
| - Simple sentence structure |
| - Common verbs and nouns |
| - Basic grammar patterns |
| - Repetition and reinforcement |
|
|
| The dataset intentionally avoids: |
| - Instruction-following |
| - Dialog formatting |
| - Reasoning traces |
|
|
| This design allows clear observation of **language emergence** in small models. |
|
|
| --- |
|
|
| ## Training Procedure |
|
|
| - Optimizer: Adam |
| - Learning rate: 3e-4 |
| - Sequence length: 64 |
| - Epochs: 12000 |
| - Loss function: Cross-Entropy Loss |
| - CPU - AMD Ryzen 5 5600G 32GB (0.72 TFLOPS) |
|
|
| Training includes checkpointing and continuation support. |
|
|
| --- |
|
|
| ## Intended Use |
|
|
| ### Supported Use Cases |
|
|
| - Educational experiments |
| - Language modeling research |
| - Studying emergent structure in small neural networks |
| - Baseline comparisons for future MiniText versions |
|
|
| ### Out-of-Scope Use Cases |
|
|
| - Conversational agents |
| - Instruction-following systems |
| - Reasoning or math tasks |
| - Production deployment |
|
|
| --- |
|
|
| ## Example Output |
|
|
| Prompt: |
| o gato é |
|
|
| Sample generation: |
| o gato é um animal |
|
|
| Note: Output quality varies due to the minimal size of the model. |
|
|
| --- |
|
|
| ## Limitations |
|
|
| - Limited vocabulary and coherence |
| - No reasoning or factual understanding |
| - Susceptible to repetition and noise |
| - Not aligned or safety-tuned |
|
|
| These limitations are **expected and intentional**. |
|
|
| --- |
|
|
| ## Ethical Considerations |
|
|
| This model does not include safety filtering or alignment mechanisms. |
| It should not be used in applications involving sensitive or high-risk domains. |
|
|
| --- |
|
|
| ## Future Work |
|
|
| Planned extensions of the MiniText family include: |
|
|
| - MiniText-v1.1-Lang (improved Portuguese fluency) |
| - MiniText-Math (symbolic pattern learning) |
| - MiniText-Chat (conversation fine-tuning) |
| - MiniText-Reasoning (structured token experiments) |
|
|
| Each version will remain linked to this base model. |
|
|
| --- |
|
|
| ## Citation |
|
|
| If you use MiniText-v1.0 in research or educational material, please cite the project repository. |
|
|
| --- |
|
|
| ## License |
|
|
| MIT License |
|
|
|
|
| Made by: Arthur Samuel(loboGOAT) |
|
|
