Switch from getrlimit(RLIMIT_AS) to android_mallopt(M_SET_ALLOCATION_LIMIT_BYTES).
The RLIMIT_AS limit was introduced as a security mitigation, but it
isn't exactly what the media processes want to control. It is also
problematic under sanitizers which allocate large amounts of address
space as shadow memory, and is especially problematic under shadow
call stack, which requires 16MB of address space per thread. Instead,
use the newly introduced android_mallopt(M_SET_ALLOCATION_LIMIT_BYTES)
to control the allocator's memory limit directly.
Also remove ASAN/HWASAN/CFI specific hacks; they are no longer
necessary because these tools consume address space using mmap and
not the allocator, and remove the 64-bit pointer check before calling
__scudo_set_rss_limit, since otherwise the limit would stop being
enforced in 32-bit mode with Scudo.
Bug: 118642754
Change-Id: Ie66128626976c0b04d5dafd455c375bbfdccc083
5 files changed