Most AVR compilers divide RAM stack into 2 sections - 1) Data Stack 2) Return Stack
Data stack stores the the local(auto for c) variables within a function/block. These are accessed by y-register.
On the other hand Return stack saves the return address when a function call/interrupt is executed. This is accessed internall by CPU using SPH/L registers & requires 2/3 byes depending on the AVR in use. 'Return Stack Size' specfies no. of byes/words set aside by compiler for calls/interrupt nesting etc.
e.g. If you specify return stack size=30 bytes for m8, then you can nest upto 15 function calls/interrupts.
If you want to know what actually this does - During startup compiler's init routine,
1) SPH:L = Last address of RAM
2) Y = (Last address of RAM) - Return Stack Size
Now when you call a function or interrupt occurres, SPH:L decreamented by 2/3 bytes automatically by CPU & increamented on return from function/interrupt.
When you declare a local variable, Y is decreamented by sizeof(var) by compiler inserted code & can be accessed by y-specific addressing modes. Just before returning from the function, compiler again inserts the code to re-increament the Y.
As in any stack, it must be balanced at entry/exit, complier takes care of that. Another major concern is stack overflow/underflow, that means,
1) you nest more calls than your specified return stack size or
2) you tried to return when there is nothing on return stack or
3) you declare more local data than your data stack or
3) you tried to access the local data when there is none.
Some compilers provide libs to check such conditions & allow to take corrective measure.
Take care that each compiler specify return stack size differently. For ICCAVR it is in 'bytes', for IAR it is no. of nesting levels. Read before you specify.
I hope it is clear.
If you have any doubts I'll be glad to clear them.