The overlay technique is a means to side-step memory limitations on severely RAM constrained systems. It does this by dedicating a segment of memory to multiple routines and overwriting the segment with the required code on an as-needed basis.
The use of overlays can be a source of frustration for anyone engaged in reverse engineering or source code archaeology because the same memory addresses are called by very different routines for conflicting purposes. Perhaps the most well known program on the C64 that makes extensive use of overlays is GEOS. An overlay should not be confused with other technologies that serve similar ends.
An overlay is a specific form of self-modifying code but not all forms of self-modifying code use overlays. Unlike virtual memory memory or even a primitive scratch file, data is not written to disk. Rather, a required segment of code is loaded from external storage on-demand at the expense of some other routine in system memory. The external storage is usually disk, but this is not always the case. An REU or even ROM can just as easily host overlay segments as a floppy disk or hard disk. All that is required is the storage device be able to perform random access efficiently.
Obviously, care must be taken to ensure that the proper code segment is in system memory when it is needed. To this end nearly all implementations of overlays use have at least the following features: First, a variable to identify which segment of code is currently loaded into RAM. All code sensitive to the code in that segment must necessarily check the status of this variable before executing. Second, a mechanism must exist to force an overlay to occur if code is about to be executed that requires a different routine than the one that is currently in memory.