A subroutine is a part of a program, which perfoms some function needed (usually) several different other places in the program: Rather than repeating the same functionality once every time it is needed, other parts of the subroutine may call then entire subroutine using a single command. When such a call is done, the computer makes a "note" of where it "came from" before turnig its attention to execute the code in the subroutine. The subroutine ends in a special command that instructs the computer to jump back to the location noted earlier, in order to continue execution back in the "calling" part of the program.
This concept not only saves space, it also allows for a lot of complex constructs, such as recursive coding that would be much more difficult without the use of subroutines.
Very early implementations of the subroutine concept only had space for a single return address, which meant that subroutines could not themselves call another subroutine, since this would require more than one return address. The C64 uses the CPU stack to hold several return addresses, allowing for subroutines calling further subroutines. This allows for a highly ordered programming "style", in which common tasks of all "sizes" and complexities are divided into self-contained subroutines calling each other.
Subroutines in BASIC
In Commodore BASIC V2, as in most implementations of BASIC, a subroutine call is initiated with the GOSUB command: This places a total of five bytes on the stack; two bytes for a pointer to the address in BASIC program memory from where the call is performed, along with two more bytes holding the BASIC line number, and finally the value 141/$8D as a "marker" to indicate the location of this information. This value "happens" to be the token code for the GOSUB keyword.
The BASIC subroutine should end with the RETURN command, which first checks to see if the topmost value on the stack is the "marker" byte (141/$8D) of a GOSUB call — if this is not the case, a ?RETURN WITHOUT GOSUB ERROR IN line occurs. Otherwise, it pulls the other four bytes from the stack to restore the pointers, i.e. "send" BASIC back to the GOSUB command it "came from". The RETURN handler routine falls through to the routine for handling DATA statements (which effectively simply skips the current BASIC statement and moves the pointers to the next one) in order to get to the statement following the initial GOSUB statement.
Since the stack is 256 bytes "deep", and each BASIC subroutine call takes up 5 bytes, the theoretical limit is at 51 "levels", not including the "mainline" program that initiated the first of fifty-one GOSUB's. But the stack is also used for other purposes, such as FOR-NEXT-loops, and when the system responds to interrupts. This still leaves room for dozens of levels, so it is unlikely that the average BASIC programmer will run into the ?OUT OF MEMORY ERROR IN line that appears if the limits are exceeded.
Subroutines in machine language
The CPU itself supports subroutines, which also make use of the stack: A subroutine is initiated with the JSR instruction; a three-byte instruction which first takes its own location (address) in memory, adds two to this location, and then stores the result as two bytes on the stack. Upon reaching an RTS instruction, the CPU pulls back the two-byte address and increases it by one address: Together with the two locations added when the JSR was executed, the return address has been increased by a total of three bytes; exactly the length of the JSR command, causing the CPU to continue at the instruction following the JSR as it returns from the subroutine.
Each subroutine call taking two of the 256 bytes available, the stack in theory allows for a total of 128 nested subroutines, but this maximum again rules out all the other uses of the stack, like responding to interrupts.
When calling machine language programs using the SYS command, the user-supplied address is called as a subroutine using what equates to a subroutine call: This is why an RTS at the "end" of the machine language routine seems to "magically" hand the system back to BASIC and its READY. prompt.