Use specific RSS check under 64-bit Scudo
Scudo is a hardened memory allocator that can be enabled on a per target basis,
and replaces malloc/free/etc with its own (go/scudo-allocator). It lives in
LLVM's compiler-rt project, along with ASan and the other sanitizers.
The 64-bit version of the allocator reserves (but doesn't commit) a large
portion of the address space, which is not compatible with the address space
limit enforced by RLIMIT_AS.
Scudo offers a function to enforce an upper RSS limit (note this is the actual
RSS of a process, not its AS), use it if running with 64-bit Scudo.
Bug: 63907455
Test: aosp compiled & run in emulator
Test: manually enabling Scudo for mediaextractor & mediacodec, flashing & using
a marlin
Change-Id: Iaef6ff04077b22d16b0b62d413081ee8e6ba6ea0
diff --git a/media/libmedia/MediaUtils.cpp b/media/libmedia/MediaUtils.cpp
index 320c7a9..bcc7ebf 100644
--- a/media/libmedia/MediaUtils.cpp
+++ b/media/libmedia/MediaUtils.cpp
@@ -25,6 +25,7 @@
#include "MediaUtils.h"
extern "C" size_t __cfi_shadow_size();
+extern "C" void __scudo_set_rss_limit(size_t, int) __attribute__((weak));
namespace android {
@@ -65,6 +66,14 @@
maxMem = propVal;
}
+ // If 64-bit Scudo is in use, enforce the hard RSS limit (in MB).
+ if (maxMem != SIZE_MAX && sizeof(void *) == 8 &&
+ &__scudo_set_rss_limit != 0) {
+ __scudo_set_rss_limit(maxMem >> 20, 1);
+ ALOGV("Scudo hard RSS limit set to %zu MB", maxMem >> 20);
+ return;
+ }
+
// Increase by the size of the CFI shadow mapping. Most of the shadow is not
// backed with physical pages, and it is possible for the result to be
// higher than total physical memory. This is fine for RLIMIT_AS.