mini-AGI Overview

The mini-AGI model comprises 540.1 million total parameters, including 531.6 million expert parameters distributed across 169 experts, each with 3.15 million parameters. It requires a minimum of 8 GB VRAM, with the RTX 3070 serving as a reference GPU. The model is implemented in Python (version 3.10 or newer) and utilizes the Adam optimizer.

Training Details

The model's context window size is 4,096 characters, allowing for a maximum of 26 block applications per character and 24 recurrent block applications. The training configuration employs a trunk learning rate set at 0.1 times the expert learning rate. Currently, the model has read 318.1 million characters, with weights not yet published as it completes its first pass over the training corpus. Expected completion of training is projected to take a couple of weeks at the current rate.

Performance Metrics

Held-out loss metrics indicate varying performance across different tasks: chess at 0.552 nats/char, stories at 0.637 nats/char, arithmetic at 0.657 nats/char, code at 0.739 nats/char, reasoning at 0.794 nats/char, chat at 0.831 nats/char, chat_hermes at 1.178 nats/char, and Wikipedia at 1.280 nats/char. The model's ability to continually learn from user interactions is emphasized in its documentation, stating, "The model is genuinely yours: trained on your hardware, on your data, that keeps learning from every conversation you have with it."

Implications for Practitioners

Developers can leverage mini-AGI to create personalized models that adapt to individual user data, enhancing the relevance and accuracy of AI interactions. This follows a trend in AI personalization that has been gaining traction, as seen in recent advancements across the industry.