Exec Shield is a project started at Red Hat, Inc in late 2002 with the aim of reducing the risk of worm or other automated remote attacks on Linux systems. The first result of the project was a security patch for the Linux kernel that emulates an NX bit on x86 CPUs that lack a native NX implementation in hardware. While the Exec Shield project has had many other components, some people refer to this first patch as Exec Shield.
The first Exec Shield patch attempts to flag data memory as non-executable and program memory as non-writeable. This suppresses many security exploits, such as those stemming from buffer overflows and other techniques relying on overwriting data and inserting code into those structures. Exec Shield also supplies some address space layout randomization for the mmap and heap base.
The patch additionally increases the difficulty of inserting and executing shellcode, rendering most exploits ineffective. No application recompilation is necessary to fully utilize exec-shield, although some applications (Mono, Wine, XEmacs, Mplayer) are not fully compatible.
Other features that came out of the Exec Shield project were the Position Independent Executables (PIE), the address space randomization patch for Linux kernels, a wide set of glibc internal security checks that make heap and format string exploits near impossible, the GCC Fortify Source feature, and the port and merge of the GCC stack-protector feature.
Exec Shield works on all x86 CPUs utilizing the Code Segment limit. Because of the way Exec Shield works, it is very lightweight; however, it won't fully protect arbitrary virtual memory layouts. If the CS limit is raised, for example by calling mprotect to make higher memory executable, then the protections are lost below that limit. Ingo Molnar points this out in an e-mail conversation. Most applications are fairly sane at this; the stack (the important part) at least winds up above any mapped libraries, so does not become executable except by explicit calls by the application.
As of August, 2004, nothing from the Exec Shield projects attempt to enforce memory protections by restricting mprotect on any architecture; although memory may not initially be executable, it may become executable later, so the kernel will allow an application to mark memory pages as both writable and executable at the same time. However, in cooperation with the Security-Enhanced Linux project (SELinux), the standard policy for the Fedora Core distribution does prohibit this behavior for most executables, with only a few exceptions for compatibility reasons.
Exec Shield was developed by various people at Red Hat; the first patch was released by Ingo Molnar of Red Hat and first released in May 2003. It is part of Fedora Core 1 through 6 and Red Hat Enterprise Linux since version 3.[1] [2] Other people involved include Jakub Jelínek, Ulrich Drepper, Richard Henderson and Arjan van de Ven.
Molnar commented in 2007 on LWN.net that "bits of [exec-shield] went upstream, but a fair chunk didn't."[3]