Use specific RSS check under 64-bit Scudo

Scudo is a hardened memory allocator that can be enabled on a per target basis,
and replaces malloc/free/etc with its own (go/scudo-allocator). It lives in
LLVM's compiler-rt project, along with ASan and the other sanitizers.

The 64-bit version of the allocator reserves (but doesn't commit) a large
portion of the address space, which is not compatible with the address space
limit enforced by RLIMIT_AS.

Scudo offers a function to enforce an upper RSS limit (note this is the actual
RSS of a process, not its AS), use it if running with 64-bit Scudo.

Bug: 63907455
Test: aosp compiled & run in emulator
Test: manually enabling Scudo for mediaextractor & mediacodec, flashing & using
      a marlin

Change-Id: Iaef6ff04077b22d16b0b62d413081ee8e6ba6ea0
1 file changed