PCKMIND · SMALL LANGUAGE MODELMy third language model,
PckMind.
I’ve just started work on my third language model, PckMind. I’ll share the development process, experiments and progress here.
PckMindMODEL IN DEVELOPMENT · WEB CHAT NOT READYThis web preview is not connected to the PckMind model. You can see its actual output in the screenshot below.
How much can a 64K-parameter model say?.
PckMind is the third small language model I’ve developed. I use my first model, PckBot, for my own automation work; the second was trained for a company. I plan to release PckMind publicly. Built with PyTorch, this GPT-style decoder-only Transformer can currently generate short text, but it cannot yet hold a coherent conversation. Sometimes it even sounds like bird language 🐦😅 I’m sharing these outputs to show the development process as it is and how the model changes over time.
This first version is a working but limited starting point. In future versions, I want to improve the architecture and training data so the model can produce more meaningful, consistent and longer responses.
- Trainable parameters
- 63,949 · about 64K
- Architecture
- Decoder-only Transformer
- Transformer blocks
- 1
- Attention heads
- 4
- Embedding size
- 64
- Context length
- 64 tokens
- Vocabulary
- 77 characters
- Training data
- 62,852 characters
- Tokenizer
- Character-level
- Attention
- Causal self-attention
- Optimizer
- AdamW
- Framework
- PyTorch
More Turkish, a stronger model.
For the next update, I plan to increase the parameter count and retrain the model on a broader Turkish dataset. The goal is a language model that understands Turkish better and responds more naturally. I’ll share progress and new experiments here.