Building Autonomous ML Research Loops with Karpathy’s AutoResearch Framework

How to Build an Autonomous Machine Learning Research Loop in Google Colab Using Andrej Karpathy’s AutoResearch Framework for Hyperparameter Discovery and Experiment Tracking

Andrej Karpathy’s AutoResearch framework enables the creation of automated experimentation pipelines that programmatically modify training configurations. The system evaluates model performance using the validation bits-per-byte (val_bpb) metric to autonomously identify superior hyperparameter sets.

Why This Matters

Manual hyperparameter tuning is a significant bottleneck in machine learning research, often requiring constant human intervention and specialized infrastructure. This framework democratizes autonomous research by allowing engineers to run iterative training loops in lightweight environments like Google Colab, shifting the focus from manual adjustment to high-level experiment design. By automating the modification of training scripts and the evaluation of results, researchers can explore a broader search space of architectural and optimization parameters without the cost of dedicated hardware management.

Key Insights

• Automated environment setup using pip and git to clone the autoresearch repository directly into Google Colab (2026).
• Dynamic configuration patching of train.py and prepare.py to fit experiments within Colab’s resource constraints, such as reducing MAX_SEQ_LEN to 512.
• Establishment of a baseline performance metric using val_bpb (validation bits-per-byte) to serve as a reference point for all subsequent iterations.
• Programmatic hyperparameter discovery through a defined search space including WINDOW_PATTERN, TOTAL_BATCH_SIZE, and various learning rates.
• Iterative model improvement where the system ‘keeps’ configurations that lower the val_bpb and ‘discards’ those that fail to exceed the current best.

Working Examples

Initial environment setup and repository cloning for the AutoResearch framework.

import os, sys, subprocess, json, re, random, shutil, time
from pathlib import Path
def pip_install(pkg):
    subprocess.check_call([sys.executable, "-m", "pip", "install", "-q", pkg])
for pkg in ["numpy","pandas","pyarrow","requests","rustbpe","tiktoken","openai"]:
    try:
        __import__(pkg)
    except:
        pip_install(pkg)
import pandas as pd
if not Path("autoresearch").exists():
    subprocess.run(["git","clone","https://github.com/karpathy/autoresearch.git"])
os.chdir("autoresearch")

Functions for sampling new hyperparameter candidates and executing automated training runs.

def sample_candidate():
    keys=random.sample(list(SEARCH_SPACE.keys()),random.choice([2,3,4]))
    cand=dict(base_hparams)
    changes={}
    for k in keys:
        cand[k]=random.choice(SEARCH_SPACE[k])
        changes[k]=cand[k]
    return cand,changes

def run_experiment(tag):
    log=f"{tag}.log"
    subprocess.run(f"python train.py > {log} 2>&1",shell=True)
    metrics=parse_run_log(log)
    metrics["log"]=log
    return metrics

Practical Applications

• Use Case: Autonomous hyperparameter optimization for language models where the system iteratively tests learning rates and batch sizes to minimize validation loss.
• Pitfall: Inadequate resource management in cloud notebooks; failing to adjust DEVICE_BATCH_SIZE or TIME_BUDGET can lead to out-of-memory errors or session timeouts.
• Use Case: Automated experiment logging using results.tsv to maintain a structured history of all trials, enabling easy comparison of architectural changes.
• Pitfall: Over-reliance on random sampling without constraints; testing incompatible hyperparameter combinations can waste computational budget on invalid training runs.

References:

• https://www.marktechpost.com/2026/03/12/how-to-build-an-autonomous-machine-learning-research-loop-in-google-colab-using-andrej-karpathys-autoresearch-framework-for-hyperparameter-discovery-and-experiment-tracking/
• https://github.com/karpathy/autoresearch.git