I have though of a reasonably simple way to improve control flow security with an already existing ISA. I’m noting it down for my own future reference.
For this to work, you need to find an instruction that has the following properties:
- Is reasonably long.
- Is uncommon in code.
- Is unlikely to appear in the middle of instruction opcodes.
- Is considered a NOP on the existing ISA.
The processor would then verify that this is the instruction executed after a function returns, thus limiting return points to only the places with such instructions.
This can also be emulated by software but in that case a better approach can be used which I’ll cover below.
This can be significantly improved if the processor uses some homomorphic encryption on call and ret to encrypt and decrypt the return address with a key contained in a special register, as this would significantly decrease the likelihood of an attacker learning the encrypted address for a specific snippet for that process.
When emulating these methods with software, the resulting “instruction” can contain an identifier of the called function, this would be verified on return and ignored by the returning function by adding the identifier length to the instruction pointer before returning. This would make it impossible to return to anywhere other than places containing the specified return string. I suspect this is what the RAP plugin by the PAX Team does but I can’t say for sure as I never sat down to read the code. This may be a problem with things like function pointers as you may not know the calling function’s code before hand but there may be ways to work around this problem. This can also be combined with homomorphic cryptography to make attacks significantly harder as the function identifier can be used as encryption key.
For now I’ll just document this, maybe I’ll get it implemented on the future. We’ll see.