Implementation of nested functions

Question

GCC uses something called a trampoline.

Information: http://gcc.gnu.org/onlinedocs/gccint/Trampolines.html

A trampoline is a piece of code that GCC creates in the stack to use when you need a pointer to a nested function. In your code, the trampoline is necessary because you pass g as a parameter to a function call. A trampoline initializes some registers so that the nested function can refer to variables in the outer function, then it jumps to the nested function itself. Trampolines are very small — you “bounce” off a trampoline and into the body of the nested function.

Using nested functions this way requires an executable stack, which is discouraged these days. There is not really any way around it.

Dissection of a trampoline:

Here is an example of a nested function in GCC’s extended C:

void func(int (*param)(int));

void outer(int x)
{
    int nested(int y)
    {
        // If x is not used somewhere in here,
        // then the function will be "lifted" into
        // a normal, non-nested function.
        return x + y;
    }
    func(nested);
}

It’s very simple so we can see how it works. Here is the resulting assembly of outer, minus some stuff:

subq    $40, %rsp
movl    $nested.1594, %edx
movl    %edi, (%rsp)
leaq    4(%rsp), %rdi
movw    $-17599, 4(%rsp)
movq    %rsp, 8(%rdi)
movl    %edx, 2(%rdi)
movw    $-17847, 6(%rdi)
movw    $-183, 16(%rdi)
movb    $-29, 18(%rdi)
call    func
addq    $40, %rsp
ret

You’ll notice that most of what it does is write registers and constants to the stack. We can follow along, and find that at SP+4 it places a 19 byte object with the following data (in GAS syntax):

.word -17599
.int $nested.1594
.word -17847
.quad %rsp
.word -183
.byte -29

This is easy enough to run through a disassembler. Suppose that $nested.1594 is 0x01234567 and %rsp is 0x0123456789abcdef. The resulting disassembly, provided by objdump, is:

   0:   41 bb 67 45 23 01       mov    $0x1234567,%r11d
   6:   49 ba ef cd ab 89 67    mov    $0x123456789abcdef,%r10
   d:   45 23 01 
  10:   49 ff e3                rex.WB jmpq   *%r11

So, the trampoline loads the outer function’s stack pointer into %r10 and jumps to the nested function’s body. The nested function body looks like this:

movl    (%r10), %eax
addl    %edi, %eax
ret

As you can see, the nested function uses %r10 to access the outer function’s variables.

Of course, it’s fairly silly that the trampoline is larger than the nested function itself. You could easily do better. But not very many people use this feature, and this way, the trampoline can stay the same size (19 bytes) no matter how large the nested function is.

Final note: At the bottom of the assembly, there is a final directive:

.section        .note.GNU-stack,"x",@progbits

This instructs the linker to mark the stack as executable.

Leave a Comment Cancel reply