Self-editing models
Can a model have general tools to read, write and visualize its own weights as much as it wants?
Self-editing LLM
It has to work with small models locally, on specific benchmarks like nanoGPT, and show that the approach scales to big models.
It should have general tools to read, write and visualize its own weights as much as it wants. This can be fully flexible through Python interaction with the model file, just like a human would.
It could for example run (versioned) fine-tuning on its own layers, or attach new randomly initialized layers or blocks and train them.
There has to be a cost component to this, as running full fine-tuning is very expensive and the model has to weigh whether it is a worthwhile experiment. Everything should be compute relative/optimal. But if it simulates a good ML researcher it should naturally start with small changes and toy experiments to build confidence.