Sunday, July 3, 2011

Alert: vsftpd download backdoored

[With thanks to Mathias Kresin for being the first to notice]

An incident, what fun! Earlier today, I was alerted that a vsftpd download from the master site (vsftpd-2.3.4.tar.gz) appeared to contain a backdoor:

The bad tarball is (sha256sum):

2a4bb16562e0d594c37b4dd3b426cb012aa8457151d4718a5abd226cef9be3a5 vsftpd-2.3.4.tar.gz

And, of course, the GPG signature notices:

$ gpg ./vsftpd-2.3.4.tar.gz.asc
gpg: Signature made Tue 15 Feb 2011 02:38:11 PM PST using DSA key ID 3C0E751C
gpg: BAD signature from "Chris Evans <>"

Check your signatures :)

Ideally, you'll see something like:

gpg: Signature made Tue 15 Feb 2011 02:38:11 PM PST using DSA key ID 3C0E751C
gpg: Good signature from "Chris Evans <>"
Primary key fingerprint: 8660 FD32 91B1 84CD BC2F 6418 AA62 EC46 3C0E 751C

Signatures aside, I also took the liberty of moving most of the vsftpd site and latest download to a hosting provider I have more faith in:

The backdoor payload is interesting. In response to a :) smiley face in the FTP username, a TCP callback shell is attempted. There is no obfuscation. More interestingly, there's no attempt to broadcast any notification of installation of the bad package. So it's unclear how victims would be identified; and also pretty much guaranteed that any major redistributor would notice the badness. Therefore, perhaps someone was just having some lulz instead of seriously trying to cause trouble.

Friday, May 27, 2011

libxml vulnerability and interesting integer issues

A while ago, I was playing with grammar-based XPath fuzzing and I found and fixed an interesting libxml bug. The commit, for the curious, is here:

The trigger for this bug was the XPath expression: //@*/preceding::node()/ancestor::node()/ancestor::foo['foo'] which for some reason I haven't yet analyzed leads to a pathologically large collection of nodes within libxml.

As the nodeset is grown and grown, things get interesting when the system runs out of memory whilst trying to double the size of the nodeset:

cur->nodeMax *= 2;
temp = (xmlNodePtr *) xmlRealloc(cur->nodeTab, cur->nodeMax *
if (temp == NULL) {
xmlXPathErrMemory(NULL, "growing nodeset\n");
cur->nodeTab = temp;

Notice how the max number of allocated nodes in the "cur" structure is doubled before doing, checking and assigning the reallocation. This means that in the event of a realloc() failure, one of two things will happen:
  • If you malloc() implementation exits the process upon alloc failure (such as Chromium), lucky you! You dodged a bullet.

  • More typically, you'll exit this function with "cur" in an inconsistent state, i.e. it indicates it can hold more data than it really can.
Despite the call to xmlXPathErrMemory(), XPath processing continues with "cur" in an inconsistent state, leading to a heap-based buffer overflow.

The fix is to only update the structure's "max size" member after a successful expansion of the underlying array. libxml already did this in most places, using a paradigm such as:

temp = (xmlNodePtr *) xmlRealloc(cur->nodeTab, cur->nodeMax * 2 *
if (temp == NULL) {
xmlXPathErrMemory(NULL, "growing nodeset\n");
cur->nodeMax *= 2;
cur->nodeTab = temp;

... which leads us nicely into part 2:

Interesting integer issues

Even with the fix applied, there are some really interesting integer issues going on here. The astute will have noticed a possible integer overflow in the argument to xmlRealloc(), which is a general hazard with the common pattern of "double the size of the array if we ran out of room". Digging in to more detail, we see a fascinating difference in behaviour between 32-bit and 64-bit builds and maybe even learn a thing or two about integer promotion rules.

On 32-bit builds

First, let's quickly note that cur->nodeMax is of type int. That's likely the wrong type for a couple of reasons, but we will run with it for our analysis.
On 32-bit, sizeof(int) == sizeof(size_t) so we're looking at a fairly basic possible integer overflow. But can it ever be triggered? Does the 32-bit address space offer enough room?
The case with the least space requirements for the 32-bit address space is the case where we already have a 2GB (2^31) allocation and are attempting to double it -- leading to an integer overflow at 2^32 and an attempt to realloc() to 0 bytes.
But to arrive at the 2^31 allocation, we need to have a 1GB -> 2GB realloc() succeed first.
This is actually unlikely on 32-bit Linux -- which typically has just the lower 3GB of address space usable. 1GB + 2GB + program text etc. won't simultaneously fit, so the only way 1GB -> 2GB realloc() will succeed is if the allocation can be expanded in-place. glibc malloc() will typically use mmap() for large allocations, and put them towards the top of address space in order to reserve room for standard heap expansion. So realloc() on such an mmap()ed chunk typically won't have room to mremap() to a larger size.
All in all, this seems to be a hard-to-exploit bug, unless the attacker has a lot of control over other allocations (to influence the placement of the 1GB mmap() lower down in the address space and then free() whatever chunks were needed to do that, before the realloc()). Further research might be merited for other allocators (e.g. tcmalloc) and 32-bit-process-on-64-bit-host (>3GB address space available).

On 64-bit builds

With 64-bit, we have a much larger address space. This means that we should be able to successfully allocate the growing array -- and the objects contained within it -- right up until the fault condition.

The fault condition is that cur->nodeMax * 2 will eventually become negative. This negative int is then multiplied by sizeof(xmlNodePtr). Remember that sizeof() returns a size_t, so we have an int -> size_t promotion at this point. This will result in sign-extension to a 64-bit size, which will end up massive and the system allocator will not be able to satisfy an allocation of that size. Therefore, a lucky lack of impact on 64-bit.

Finally, note that there's a real subtlety in the ordering of the expression given to xmlMalloc():

cur->nodeMax * 2 * sizeof(xmlNodePtr)

Things would be very different if we had:

cur->nodeMax * sizeof(xmlNodePtr) * 2

Promotions for left-to-right operator sequences are done in strict left-to-right ordering, as opposed to some kind of overall max(precedence) promotion. Therefore, in this latter case, the initial negative-related failure will be avoided; we're looking at:

(int) 2^30 -> (int) -(2^31) -> (size_t) 0xffffffff80000000 * 8 == (size_t) 0xfffffffc00000000
(int) 2^30 -> (size_t) 2^30 * 8 -> (size_t) 2^30 * 8 * 2 == (size_t) 2^34

Seeing that nodeNr is just an int, we would likely see subsequent memory corruption if nodeNr goes subsequently negative after its increment in a statement such as cur->nodeTab[cur->nodeNr++] = xmlXPathNodeSetDupNs(node, ns);

Wednesday, May 25, 2011

Bug bounties vs. black (& grey) markets

I'm just back from the fun that was HiTB Amsterdam 2011. (Plug: you should check out one of the HiTB series if you haven't yet; Dhillon and crew invariably put a good, intimate conf together).

I sat on the day 2 keynote panel on "The economics of vulnerabilities". As usual, talking about this topic was great fun and the audience asked some great questions. Predictably, the topic strayed on to black market sales as an interesting sub-discussion. With 6 people on the panel, it was hard to cover this in the detail it deserves, and I think a few important subtleties were missed. I'll try to cover some of them here.

Vulnerability reward programs do not "buy" bugs, nor do they aim to compete with the black market
Remember that the black (or grey) markets buy exploits, not vulnerabilities. The latter are just the first step towards exploits, which are hard to write on modern software. Also remember that reward programs are not buying even vulnerabilities. Typically, they are a "thank you" mechanism for talented researchers who used their skills to make things better.
Also, there's often a separation of which researchers participate where, along ethics lines; see below.

A vulnerability reward program will indirectly compete with black market sales
It's interesting to note that a reward program doesn't have to outbid dubious markets in order to have a benefit in this area. These days, there's a lot of independent rediscovery of the same vulnerabilities -- ZDI quoted 22 "collisions" for the most recent year.
So any motivation you can provide for white hats to discover vulnerabilities will inevitably kill the occasional black market vulnerability.
A quick story in support: the WebKit vulnerability used by VUPEN to pwn Safari at this years' pwn2own competition was independently reported to the Chromium project by researcher Martin Barbella. Thanks to Inferno's lightning quick fix, Chrome entered pwn2own without that bug. Martin, of course, received a $1000 Chromium Security Reward (on top of all his others).

Black / grey market sales are a dangerous alternative to consider
Each researcher has to set their own ethics, of course. Hopefully, most of us get into this industry to make users safer and software more secure. Aside from reward programs and ZDI, there's also a large number of well-paid security jobs sponsored by corporations, so no need to start selling exploits to feed the family.
If you sell an exploit to someone, it's basically going to be used to exploit end users of the software. This could harm a lot of people if the target is mass malware for financial gain. Or it could seriously harm some targeted individuals if a government of dubious human rights commitment gets their hands on it.

"Credit", whilst important, is not a full replacement for a monetary reward
To be clear about it: if you launch a vulnerability reward program, you will receive more vulnerability reports from a wider range of researchers. The power of credit and prestige is often cited as an argument to not launch a reward program, but the fact remains that you will get more reports if you have a program in place. And as long as you have a culture of fixing security bugs promptly, your users will be safer thanks to having a reward program.

Wednesday, April 27, 2011

Fiddling with Chromium's new certificate pinning

Over the past few years, there have been various high-profile incidents and concerns with the Certificate Authority-based infrastructure that underpins https connections. Various different efforts are underway to tackle the problem; many are enumerated here:

And in terms of things baked directly into the browser, we have things like Firefox's Certificate Patrol add-on:

My colleague Adam Langley summarized some features and directions we've been exploring in Chromium recently, it's a good read:

These features can also be controlled via the command-line, so to give a glimpse of the future, I present to you:

Twitter Like A Boss

Run Chrome (v12 dev channel or newer required) with a command line like this:

google-chrome --user-data-dir=/tmp/chrome_twitter --incognito --disable-plugins --proxy-server=localhost:1 --proxy-bypass-list=,https://*,https://* --hsts-hosts='{"df0sSkr4gOg4VK8d/NNTAWFtAN/MjCgPCJ5ml+ucdZE=":{"expiry":2000000000.0,"include_subdomains":true,"mode":"strict","public_key_hashes":["sha1/TXoScD1SXPfhmRO8ACTPrkXD9Yk="]},"tGm+XsbBPK211uMWtg2k071vijQkuVLvd62QzfNFol8=":{"expiry":2000000000.0,"include_subdomains":true,"mode":"strict","public_key_hashes":["sha1/06curQTaPH4PGumbNSeL79da23s="]},"wZU3atDOXaxKkaRgSdlWwB4UYjulRq46SGnIBij5I98=":{"expiry":2000000000.0,"include_subdomains":true,"mode":"strict","public_key_hashes":["sha1/O6hykhOmHJ5HQUREC0DTDeu6+mE="]}}' --user-agent='LIKE A BOSS'

(You'll need to edit a couple of things such as the command name and the temp directory if you're on Windows or Mac).

If you wish to connect securely to Twitter, well it pretty much does so... like a boss. It does the following things and defends against the following situations:

  • The --user-data-dir flag loads Twitter in a new profile so that you get a new Chrome instance and therefore new cookie jar. Therefore, carelessly clicked links in your other browsing windows won't get you XSSed.

  • The --incognito flag applies the usual incognito changes; notably, things like profile photos won't be cached to disk; might be useful if you're an activist.

  • --disable-plugins is strictly unnecessary since Twitter generally isn't using plug-ins. However, any "secure" command line should likely include that flag.

  • The --proxy-server=localhost:1 is a good defensive catch-all which will stop any site traffic being sent by your browser unless it is whitelisted. Specifically, a link to an XSS payload won't work on you. (You'll need to paste such links into an alternate browser which shouldn't be logged in to Twitter). This will also stop non-pinned https requests going out (which might otherwise compromise the integrity of the main page). Mixed-content bugs, cookie forcing and failure to mark cookies "Secure" will also be mitigated.

  • --hsts-hosts is the magic. It locks, and such that SSL traffic from/to Twitter will only be accepted if the leaf SSL certificate's public key is exactly what we expect. It's called "certificate pinning", and along with HSTS, it defends against any compromised root CA, Comodo-gate, Tunisia-like sslstrip attacks, and the "evil country owns firewall + CA" situation.

  • --user-agent='LIKE A BOSS' is strictly optional, depending on your mood.

The above command line isn't finished. Although Twitter seems to run fine, there are under-the-hood failures to and other places because I haven't added the correct certificate pin for that host. The above will break if any of the leaf certificate public keys change (this doesn't necessarily happen on expiry rollover but may otherwise happen for various reasons).

Hopefully this demo is compelling. The plan is to push this technology more and more under the covers so that it happens for less technical users who have an empty command line!

Wednesday, March 9, 2011

Multi-browser heap address leak in XSLT

It's not often that I find a bug that affects multiple different codebases in the same way, but here is an interesting info-leak bug that is currently unpatched in Firefox, Internet Explorer and Safari.

I'm releasing it now for a few reasons:
  1. The bug was already publicly noted here.

  2. This bug cannot damage anyone in and of itself; it's a low severity info-leak that does not corrupt anything. It needs to be paired with other bugs, perhaps as an exploit aid against ASLR.

  3. This is a rare and unique opportunity to directly compare vendor responses and response times for a near-identical bug. It's nice that this is a lower-severity issue as all vendors tend to treat critical issues with at least some urgency; lower severity issues serve as a better differentiator.

The bug
The bug is in the generate-id() XPath function, and is sometimes used in XSL transforms. Here's an web page that simply calls generate-id() and renders the result as a web page:

Let's see how this renders in different browsers:

Firefox (64-bit Linux)

There is no "obfuscation" that this is a raw heap address. Since Firefox is open source, we can go and look at the source code to find that indeed, the string is generated from a pointer (txXPathNodeUtils::getXSLTId):
const char gPrintfFmt[] = "id0x%016p";

Internet Explorer 8 (Windows 7)

Doesn't look like a heap address, does it? If, however, you strip off the "ID" prefix and treat the string as a [A-Z0-5] base32 encoded "little endian" string, you resolve to a nice heap address. At that address is a pointer in msxml.dll, possibly the address of a vtable for some internal xml node class.

Safari 5 (Mac OS X)

Also does not immediately look like a heap address, but libxslt is doing a simple transform on a heap address:

val = (unsigned long)((char *)cur - (char *)0);
val /= sizeof(xmlNode);
sprintf((char *)str, "id%ld", val);

These object ids bounce around all over the place. I don't know what is going on so I'm not making the claim that Opera is affected.

Latest stable Chrome (Chrome 10) is not affected. It has been removed from the "time to fix" competition in order to keep things fair.

It's on!! Who will fix it first and who will be the security laggard? Updates to be provided via Twitter: @scarybeasts

Tuesday, March 8, 2011

Busy Chrome day...

I did a bunch of fairly interesting things with my corporate hat on today (not to be confused with any of my personal research ;-)

Firstly, Chrome 10 went out with a record $16k+ series of rewards. It's continually humbling to see such a wide range of researchers and a wide range of bug categories!

Also, there are some nice new security pieces in Chrome 10. I blogged about some of these:

My personal favourite is "plug-in blocking enhancements", probably because I implemented it and am therefore biased :-) In reality, the change that's going to really help end user security is "out-of-date plug-in warnings". Users are encouraged to update to the latest security patches for their plug-ins. I personally believe this will be particularly helpful for Java, which is widely installed but users are not always the most uptodate.

And then I spoke at SANS AppSec with Adam Mein about Google's two vulnerability reward programs (Chromium and Web). This seemed to be very well received, as evidenced by the stack of insightful questions. We released a few new stats and charts, so it's probably worth me linking to the slides:

All in all a fun day!

Saturday, March 5, 2011

Dangerous file write bug in Foxit PDF Reader

This is fixed in the recently released Foxit PDF Reader v4.3.1.0218. That release is marked as an important security update, although this file bug is not mentioned.

Recently, I've been playing around with the various JavaScript APIs available in various different PDF readers. In case you wanted to do the same, I made some little tools, including a simple one to execute PDF-based JS via an URL:'hi')

The serious bug I found in Foxit PDF Reader permits arbitrary files to be written with arbitrary content, like this:'c:/autoexec.bat','echo hi mom')

Files can be overwritten as well as created.

I did some hackery on the generated PDF and managed to squeeze a full valid PDF, including simple JS payload, into 136 characters. This means I can tweet the full PoC PDF, which I will do shortly :) Here it is for completeness:

%PDF 1 0 obj<</Pages 1 0 R /OpenAction 2 0 R>> 2 0 obj<</S /JavaScript /JS (createDataObject\('c:/pwn','pwn'\))>> trailer<</Root 1 0 R>>

Thursday, February 24, 2011

I got accidental code execution via glibc?!

The story of Chromium security bug 48733, with guest Cris Neckar, part I

It has been a long time now, but the story of Chromium security bug 48733 deserves to be told. It involves intrigue in glibc and even gcc; and notably I accidentally executed arbitrary code whilst playing with this bug!

The bug was reported in July 2010, and there were instantly some WTF aspects. It caused a full browser crash on Linux, and the trigger seemed to be a long string. Such a case would tend to suggest a buffer overflow; but these are very unusual in Chromium code. Upon further investigation, the crash was occurring in the glibc function fnmatch():

int fnmatch(const char *pattern, const char *string, int flags);

And what was very strange was the trigger was not the pattern (which is a complicated string format), but simply the string itself. Further investigation narrowed the problem down to any long-ish (few megabytes+) string, if the locale was set to UTF8. A simple C test program is included at the end of the post. And here comes the killer: I was playing around and ran the program like this on my 32-bit Ubuntu 9.04 machine:

./a.out 1073741796

And accidentally achieved arbitrary code execution! The "A" characters making up the large input string actually correspond to the instruction inc %ecx so I wound up executing a bunch of those.

So what was going on?
Probably best to tackle the list of interesting points in bullet form:

  • glibc had a bug where it would use alloca() for the length of a user-supplied UTF8 string, times four (with additional integer overflow in the times four). This is good for at least a crash, because alloca() extends the stack, which is typically limited to a few MB.

  • It seems uncommon for Linux distributions to compile packages with gcc flags that defend against stack extension attacks -- more about that in part II.

  • 32-bit Ubuntu releases used to lack DEP. Perhaps they still do? This permits the execution of code contained within heap chunks, and is key to the accidental code execution achieved.

  • But how did EIP get redirected? The number passed to a.out above is a bit magic; glibc multiplies it by 4 (sizeof(wchar_t)) before passing it to alloca(), which ends up with the value 2^32 - 112. This wraps the stack pointer, causing an effective decrease in the stack of 112 bytes.

  • The decrease in stack size leads to all sorts of havoc; we're not sure, but most likely a local variable (in a subfunction of the function that called alloca()), pointing to the incoming heap string -- got plonked on top a saved EIP. I no longer have the old version of Ubuntu to test with, and more recent glibcs are fixed, so I can't confirm.

  • Note that stack extension bugs like this often sidestep a lot of system defenses, such as stack canaries (which are left undamaged) and ASLR (a valid address is automatically filled in). It's another case where Ubuntu could really have used DEP; see my older Firefox exploit for further proof!

How does part I end?
Of course, we reported the bug upstream to glibc: The somewhat terse response notes that the issue was fixed but not in which version. Because of this, no glibc security advisories were released; so apologies if your older but still supported Linux distribution might still have vulnerabilities in this area.

Although certainly not a bug in Chromium, we still paid the bug finder $1337 under the Chromium Security Reward program. We did this partly just because we can, and we love encouraging all security research. But also, we were able to work around this glibc bug in Chromium fairly trivially -- so we did so in short order. As can be seen from the Chromium bug, we had all users protected in under 20 days from the original report, despite it not being our fault!

#include <err.h>
#include <fnmatch.h>
#include <locale.h>
#include <stdlib.h>
#include <string.h>

int main(int argc, const char* argv[]) {
size_t num_as;
char* p;
setlocale(LC_ALL, "en_US.UTF8");
if (argc < 2) {
errx(1, "Missing argument.");
num_as = atoi(argv[1]);
if (num_as < 5) {
errx(1, "Need 5.");
p = malloc(num_as);
if (!p) {
errx(1, "malloc() failed.");
memset(p, 'A', num_as);
p[num_as - 1] = '\0';
p[0] = 'f';
p[1] = 'o';
p[2] = 'o';
p[3] = '.';
fnmatch("*.anim[1-9j]", p, 0);
return 0;

Wednesday, February 16, 2011

Some less obvious benefits of HSTS

HSTS, standing for HTTP Strict Transport Security, is a relatively new standard that aims to bolster the strength of HTTPS connections.

Hopefully it's about to catch on. Google Chrome has supported HSTS for a while now, and Firefox support is imminent.

The stated benefits of HSTS include:
  • Defenses against sslstrip-like attacks. The initial navigation to is automatically upgraded to HTTPS.

  • Zero tolerance for certification problems. The user is not permitted to "click through" anything such as a self-signed cert.

HSTS also comes with some less obvious benefits and security boosts, which it's worth noting:
  • Mixed-content defense. For same domain mixed-content situations, the fetches are automatically upgraded to HTTPS. This can sometimes sidestep nasty bugs.

  • Secure cookie defense. It's a pretty egregious bug for an HTTPS-only site to fail to mark its cookies "Secure", but HSTS can defend against the cookie value being sent out plaintext.

  • Cookie forcing defense. Cookie forcing is a pretty nasty MITM attack that I was playing with back in 2008. As long as HSTS is used in "includeSubDomains" mode, it can provide a defense against this subtle attack.

  • Latency win. User who navigate to or bookmark the plain HTTP are automatically bounced straight to HTTPS, without having to go via an HTTP redirect

In the future, I'm hopeful HSTS can be extended to provide defenses against possibly rogue CAs.

Wednesday, January 19, 2011

A harmless SVG + XSLT curiousity

How do you execute code in a turing complete language via the <img> tag? Why, by combining an XSL transform into an SVG image of course!

I stumbled across this old file in my archives:

If you run it e.g. in Chrome, it'll consume a load of CPU (and subsequently memory if you let it crank). I expect it'll do the same in any WebKit browser, and Opera's error message implies it has all the pieces to follow suit if I tweaked the file a bit.

It's not a significant security issue, but it's an interesting quirk. It works because SVG and XSL are both XML formats, and XSL can use a self-referential construct to operate on itself as the input document:

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="#stylesheet"?>

If the XSL output XML is valid SVG syntax, then it will render. So you can probably pull some crazy tricks to generate a complicated SVG on the fly! My sample file doesn't get that far; it simply deliberately runs an expensive stylesheet transform with a large output.

If anyone wanted to play with this, there may be interesting issues with the unusual context the XSL is executing in. What if you used xsl:import or the document() XPath function? What origin is used for security checks?, etc.