Update 14.02.2023
The information in this blog post is outdated. For more information on how to scan appliances remotely using SSH see this newer blog post.
In this blog post I’d like to outline an idea on how to perform an automated compromise assessment on Citrix Netscaler / Citrix ADC appliances.
I you haven’t heard about that vulnerability yet, you should read my tweets over the past weekend or try to get a full picture with the help of this Reddit.
This blog post is about a subsequent forensic analysis, not detection, not protection and contains no marketing blah-blah.
The described solution should work with the free / open scanners as well. They just don’t have that huge signature set as it is included in THOR.
The reason for this is that although you’ve implemented the workaround on Monday 13th of January, you can’t trust your gateway anymore.
The vulnerability is known since December 17th and the public exploit code available since Friday the 10th of January. Nation state actors have most likely used Christmas holiday season to analyze Netscaler systems in their lab and produce an exploit, just like many offensive researchers did. Since Friday the 10th, even script kiddies can exploit this vulnerbility.
Strictly speaking, you can’t trust these gateways anymore.
But how can we regain trust?
A. Wait for a patched version and reinstall the gateways from scratch
B. Perform a manual forensic analysis
C. Perform an automated forensic analysis
Automated Analysis with THOR
We don’t have or plan to release a THOR version for FreeBSD 8.4, but we could try to mount the remote filesystem of the Citrix Netscaler / ADC gateway with SSHFS.
DigitalOcean has a good tutorial on how to enable it on your workstation.
mkdir ~/netscaler-mount sudo sshfs -o allow_other,defer_permissions nsroot@citrix-gateway.nextron:/netscaler ~/netscaler-mount/
We can then use the full version of THOR to scan that drive in lab scan mode.
./thor64 --lab -p ~/netscaler-mount
The free version doesn’t have that lab scanning feature, but we can use a set of commands that achieves something very similar. (this blog post explains the differences in more detail)
./thor-linux-64 -a Filescan --intense --norescontrol --cross-platform --alldrives -p ~/netscaler-mountan
With this method we can detect dropped malicious templates.
You can use the public YARA rule, which I’ve published in LOKI’s and THOR Lite’s signature base.
Important note: this only works for templates that have been dropped since the last reboot. Users reported that dropped malicious templates get removed during a reboot. As the workaround described by Citrix requires a reboot, scanning with this single YARA rule alone won’t be a reliable method.
I recommend scanning with the full signatures set in all scanners, as they also include rules for web shells and other suspicious content in files. I’ll also extend the YARA rules on github and add further indicators.
Additional Indicators
We can use the following YARA rule to check for suspicious strings in log files. But we don’t want to use that rule on any Linux/FreeBSD system. Therefore I won’t integrate that YARA rule in the public signature base. You can just copy it from this blog post and place it in the ‘custom-signatures/yara’ sub folder (‘/signatures’ for LOKI).
This rule for the forensic artefacts is based on remarks made in this article by TrustedSec and shared as gist.
First, we mount the ‘/var/log’ directory from our remote Netscaler system.
mkdir ~/netscaler-logs sudo sshfs -o allow_other,defer_permissions nsroot@citrix-gateway.nextron:/var/log ~/netscaler-logs/
As you can see, the scanner noticed a an issued ‘whoami’ command in the ‘bash.log’ and ‘notice.log’ files, which is a good starting point for further investigations.
Final Thoughts
I think we’ll develop more rules for post-exploitation payloads over the coming days. Stay tuned and follow my twitter account for updates.
