Defining integer overflows and how hackers take advantage
An integer overflow is software behavior caused by an arithmetic operation whose numerical result is too large to store within the bit width of the system. Most machines today are either 32-bit or 64-bit. This restricts the number of bits available to store the output of an arithmetic operation to 32 or 64 bits, respectively.
Correspondingly, when an arithmetic operation produces a result that is too large to store within the bit width of the system, the result is truncated at the bit width, leading to an unexpected resultant value. This overflowed value could be used, regardless, for a critical operation such as array indexing, memory allocation or memory dereferencing.
Such behavior cannot only cause crashes in the software, but also make the software vulnerable to security exploits that deliberately exploit integer overflows to access or corrupt privileged memory in the system. The sample code below demonstrates a potential overflow in the add operation between two unsigned 32-bit values, if their sum were greater than UINT_MAX (2^32 - 1 or 0xFFFFFFFF).
In the above example, if a and b were both equal to 2^31 + 1, the resulting value of x, 2^32 + 2, would overflow 32 bits, thereby making the value of x = 2, which is (2^32 + 2) truncated to 32 bits! On line 5, this overflowed value of x is printed onto standard output. The seen result is erroneous compared to the programmer's intent of having x contain the sum of a and b. However, this overflow is benign in that it does not make this program vulnerable to attack. This is not always the case. Consider the code fragment below:
In the example above, x can still contain the overflowed value from a + b. If a and b were both 2^31 + 1, then x would be 2. If the overflowed value x were then used as the size argument to malloc, only x bytes (which is NOT equal to a + b bytes) are allocated. This creates a critical mismatch between the programmer's expectation of having allocated a + b bytes (2^32 + 2 in our example) and the system's actions of having allocated x bytes (2 in our example).
Thus, on line 7, the access p[a] (p[2^31 + 1] in our case) can access unallocated and even privileged memory locations. In particular, a malicious user might engineer the values of a and b (which are read from the user) to exploit the integer overflow and the following accesses to read or even corrupt privileged memory locations.
Also, in the example above, if the malicious user determines the address of a 2 byte memory allocation (call it L), and subsequently determines that memory representing a critical security privilege is at an offset of 40 bytes from L, the user can choose the values of a and b to be 40 and 2^32 - 38, respectively. The resultant x overflows and contains the value 2, causing a 2 byte allocation (L) on line 6. On line 7, p[a] overwrites the memory location offset at 40 bytes from L.
Such overwrites of arbitrary memory locations exploiting the integer overflow vulnerability are particularly dangerous in security-critical applications that often run with superuser privileges, due to which security-critical memory locations are within the address space of the application. In a common instance of the integer overflow vulnerability in real-world software, the attacker can overwrite the address to which the code needs to jump with the address of arbitrary code, thereby making the software execute arbitrary code.