Xander Dunn
I'm an engineer working on AI. I'm particularly interested in reasoning, code, machine language, and tool use, and embodied intelligence.
- GitHub, where my most popular repo is Skyrim modding tutorials and tools!
- A good way to see what I've been up to recently is to look at what I'm doing on GitHub.
- X
Work
- 2023+: Implemented autograd from scratch in Rust, training distributed Transformer foundation models across 8,000 GPUs and TRNs with the performance and correctness of pytorch/tensorflow/jax without the pytorch/tensorflow/jax. Data, tensor, model, pipeline, and expert parallelism. Applying to LLM self play / self improvement with code generation.
- 2020-2021: Characterizing unintended memorization (Applying Secret Sharer) and fine-tuning GPT and BERT models
- 2019-2020: WaveNet forecasting on trillions of rows of streaming music metadata
- 2017-18: "Homomorphic encryption" (signal-preserving obfuscation) using GANs
- 2015-16: Deep reinforcement learning on industrial robotics with imitation learning
I'm additionally interested in information theory, security (vulnerabilities, exploits), and trees (the plant kind, not the balanced kind). I am loosely described as a: pragmatist, realist, rationalist, empiricist, humanist, functionalist, simulationist, capitalist, techno optimist, transhumanist. I think Moravec's Paradox and the Bitter Lesson are two of the most important empirical realizations in AI.
Essays
- Thoughts on AI Training Infrastructure
- Human Popsicle
- Robotics and the Bitter Lesson
- Software Engineers Meet Bio
- Starfield Review
- On Dark Triads in Silicon Valley
- LLMs: The Important Ingredients
- Software 1.0 Security Applied to Software 2.0 Security
- AGI Non-Fiction
- AGI Sci-Fi
- Getting Up to Speed on LLMs and AGI