The British Museum algorithm is a general approach to find a solution by checking all possibilities one by one, beginning with the smallest. The term refers to a conceptual, not a practical, technique where the number of possibilities is enormous.
For instance, one may, in theory, find the smallest program that solves a particular problem in the following way: Generate all possible source codes of length one character. Check each one to see if it solves the problem. (Note: the halting problem makes this check troublesome.) If not, generate and check all programs of two characters, three characters, etc. Conceptually, this finds the smallest program, but in practice it tends to take an unacceptable amount of time (more than the lifetime of the universe, in many instances).
Similar arguments can be made to show that optimizations, theorem proving, language recognition, etc. are possible or impossible.
Newell, Shaw, and Simon called this procedure the British Museum algorithm
"... since it seemed to them as sensible as placing monkeys in front of typewriters in order to reproduce all the books in the British Museum."