So funny, was thinking of injecting “noise” some months ago to possibly improve LLM — RE digital audio sound improved by injection of noise called “dither”. Stanford paper shows that was actually insightful. The helps prevent ‘death spiral’ (the resource-wasteful reaching of massive yet insane conclusion) due to perturbations on higher / earlier hallucinations. I think of this as ‘double spot checking’ your work, or ‘faceting crazy what-ifs’ along the way — a kind of ‘self doubt, re-checking / sanity-checking, self pot-shotting’ during progress. He talks about it as “Secret #2” around 6:00 in.
