ffcall and Windows XP Data Execution Prevention

The ffcall library which is used in the GNUStep base library has a problem on Windows XP SP2 (and later) if Data Execution Prevention is turned on for all programs.

Here is a patch to ffcall 1.10, that allows the trampoline code to work with Windows Data Execution Prevention.

What is Data Execution Prevention?

In Windows Server 2003 & Windows XP SP2, Microsoft added a feature (if you have the hardware to support it), which will prevent the execution of code in areas of memory marked for data. So specifically this protects against a class of buffer overrun attacks.

I know you are saying, “So what, I don’t have self modifying code” that is what I thought too. So go and change your settings and turn on DEP. Go do it now, I’ll wait here…..

Ok, perhaps your program still runs fine, but mine didn’t. Turns out that the GNUStep implementation of NSInvocation uses the ffcall library to make the dynamic function calls. The ffcall implementaion of trampolines (basically a function pointer with context) allocates some memory and writes some opcodes in to it to set up the context state then JMP to the real function. Oops, that is self modifying code and doesn’t work any more.

The Solution: Windows doesn’t actually ban all execution in writable pages, just in data pages. Most unix OSes have similar limitations. We just need to tell Windows that we’d like a writable data page to put our code into. This is done with the VirtualAlloc function. For example:
void *addr = VirtualAlloc(NULL, bytesneeded, MEM_COMMIT, PAGE_EXECUTE_READWRITE)
Now, VirtualAlloc can only allocate multiples of the pagesize, and the bytesneeded in that call will be rounded up. So don’t be using it like malloc and calling VirtualAlloc over and over again with tiny little values. The default page size is 4k on workstation and either 4k or 2M (yes 2 megabytes!) on server platforms.

3 thoughts on “ffcall and Windows XP Data Execution Prevention”

  1. Thanks Jeremy for this information.But I have another problem regarding DEP.I am creating file using “GetTempFileName” funcion.But when I tried to export data to that file it is giving me error as “unhabdled Win32 Exception”.

  2. Sorry, I seem to have misplaced my crystal ball at the moment. I suspect this has nothing to do with DEP, but that you are reading or writing past the end of a buffer.

  3. I use Windows XP SP2, MSYS/Mingw32 toolkit. I downloaded ffcall-1.10 from http://www.haible.de/bruno/gnu/ffcall-1.10.tar.gz. I am unable to apply the patch. I get the following error message

    ————— ————————————-
    $ patch trampoline.c ffcall.diff.txt
    (Stripping trailing CRs from patch.)
    patching file trampoline.c
    Hunk #1 FAILED at 65.
    Hunk #2 FAILED at 91.
    Hunk #3 FAILED at 397.
    3 out of 3 hunks FAILED — saving rejects to file trampoline.c.rej
    (Stripping trailing CRs from patch.)
    patching file trampoline.c
    Hunk #1 FAILED at 61.
    Hunk #2 FAILED at 87.
    Hunk #3 FAILED at 379.
    3 out of 3 hunks FAILED — saving rejects to file trampoline.c.rej

Comments are closed.