mark :: blog :: security

<< prev [ 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 ] next >>

Red Hat Enterprise Linux 5.1 was released today, around 8 months since the release of 5.0 in March 2007. So let's use this opportunity to take a quick look back over the vulnerabilities and security updates we've made in that time, specifically for Red Hat Enterprise Linux 5 Server.

The graph below shows the total number of security updates issued for Red Hat Enterprise Linux 5 Server up to and including the 5.1 release, broken down by severity. I've split it into two columns, one for the packages you'd get if you did a default install, and the other if you installed every single package (which is unlikely as it would involve a bit of manual effort to select every one). So, for a given installation, the number of packages and vulnerabilities will be somewhere between the two extremes.

missing graph

So for all packages, from release up to and including 5.1, we shipped 94 updates to address 218 vulnerabilities. 7 advisories were rated critical, 36 were important, and the remaining 51 were moderate and low.

For a default install, from release up to and including 5.1, we shipped 60 updates to address 135 vulnerabilities. 7 advisories were rated critical, 26 were important, and the remaining 27 were moderate and low.

Red Hat Enterprise Linux 5 shipped with a number of security technologies designed to make it harder to exploit vulnerabilities and in some cases block exploits for certain flaw types completely. For the period of this study there were two flaws blocked that would otherwise have required critical updates:

  1. A stack buffer overflow flaw in the RPC library in Kerberos. This flaw was blocked by FORTIFY_SOURCE which removed the possibility of remote code execution. We still issued an update, as a remote attacker could trigger this flaw and cause Kerberos to crash.
  2. Another flaw in Kerberos, this time due to the free of an invalid pointer. This flaw was blocked by glibc, although a remote attacker could still cause a crash, so we issued an update.

This data is interesting to get a feel for the risk of running Enterprise Linux 5 Server, but isn't really useful for comparisons with other versions or distributions -- for example, a default install of Red Hat Enterprise 4AS did not include Firefox. You can get the results I presented above for yourself by using our public security measurement data and tools, and run your own metrics for any given Red Hat product, package set, timescales, and severities.

Although Red Hat is well known for Red Hat Enterprise Linux we actually have a large number of other supported products, both layered on top of Enterprise Linux (like Red Hat Application Stack) and stand-alone (like Red Hat Directory Server). The majority of these products are serviced through the Red Hat Network and get our security advisories in a standard way and are included in the Security Response Team metrics. But our analysis scripts were not particularly consistent in dealing with product names.

Common Platform Enumeration (CPE) is a naming scheme designed to combat these inconsistencies, and is part of the 'making security measurable' initiative from Mitre. From today we're supporting CPE in our Security Response Team metrics: we publish a mapping of Red Hat advisories to both CVE and CPE platforms (updated daily) and you can use CPE to filter the metrics. Some examples of CPE names:

cpe://redhat:enterprise_linux:5:server/firefox -- the Firefox browser package on Red Hat Enterprise Linux 5 server.
cpe://redhat:enterprise_linux:3 -- Red Hat Enterprise Linux 3
cpe://redhat/xpdf -- the xpdf package in any Red Hat product.
cpe://redhat:rhel_application_stack:1 -- Red Hat Application Stack version 1

For the past 12 months I've been keeping metrics on the types of issues that get reported to the private Apache Software Foundation security alert email address. Here's the summary for Jul 2006-Jun 2007 based on 154 reports:

User reports a security vulnerability
(this includes things later found not to be vulnerabilities)
47 (30%)
User is confused because they visited a site "powered by Apache"
(happens a lot when some phishing or spam points to a site that is taken down and replaced with the default Apache httpd page)
39 (25%)
User asks a general product support question
38 (25%)
User asks a question about old security vulnerabilities
21 (14%)
User reports being compromised, although non-ASF software was at fault
(For example through PHP, CGI, other web applications)
9 (6%)

That last one is worth restating: in the last 12 months no one who contacted the ASF security team reported a compromise that was found to be caused by ASF software.

The National Vulnerability Database provides a public severity rating for all CVE named vulnerabilities, "Low" "Medium" and "High", which they generate automatically based on the CVSS score their analysts calculate for each issue. I've been interested for some time to see how well those map to the severity ratings that Red Hat give to issues. We use the same ratings and methodology as Microsoft and others use, assigning "Critical" for things that have the ability to be remotely exploited automatically through "Important", "Moderate", to "Low".

Given a thundery Sunday afternoon I took the last 12 months of all possible vulnerabilities affecting Red Hat Enterprise Linux 4 (from 126 advisories across all components) from my metrics page and compared to NVD using their provided XML data files. The result broke down like this:

Red Hat
13% Crit 24% Important 39% Moderate 24% Low
30% High 20% Moderate 50% Low

So that looked okay on the surface; but the diagram above implies that all the issues Red Hat rated as Critical got mapped in NVD to High. But that's not actually the case, and when you look at the breakdown you get this result: (in number of vulnerabilities)

 NVD: High
23 Critical
24 Important
35 Moderate
8 Low

 NVD: Moderate
9 Crit
18 Important
22 Moderate
12 Low

 NVD: Low
7 C
32 Important
62 Moderate
51 Low

That shows nearly half of the issues that NVD rated as High actually only affected Red Hat with Moderate or Low severity. Given our policy is to fix the things that are Critical and Important the fastest (and we have a pretty impressive record for fixing critical issues), it's no wonder that recent vulnerability studies that use the NVD mapping when analysing Red Hat vulnerabilities have some significant data errors.

I wasn't actually surprised that there are so many differences: my hypothesis is that many of the errors are due to the nature of how vulnerabilities affect open source software. Take for example the Apache HTTP server. Lots of companies ship Apache in their products, but all ship different versions with different defaults on different operating systems for different architecture compiled with different compilers using different compiler options. Many Apache vulnerabilities over the years have affected different platforms in significantly different ways. We've seen an Apache vulnerability that leads to arbitrary code execution on older FreeBSD, that causes a denial of service on Windows, but that was unexploitable on Linux for example. But it has a single CVE identifier.

So if you're using a version of the Apache web server you got with your Red Hat Enterprise Linux distribution then you need to rely on Red Hat to tell you how the issue affects the version they gave you -- in the same way you rely on them to give you an update to correct the issue.

I did also spot a few instances where the CVSS score for a given vulnerability was not correctly coded. CVSS version 2 was released last week and once NVD is based on the new version I'll redo this analysis and spend more time submitting corrections to any obvious mistakes.

But in summary: for multi-vendor software the severity rating for a given vulnerability may very well be different for each vendors version. This is a level of detail that vulnerability databases such as NVD don't currently capture; so you need to be careful if you are relying on the accuracy of third party severity ratings.

It sometimes seems like me and my team are pushing security updates every day, but actually a default installation of Enterprise Linux 4 AS was vulnerable to only 3 critical security issues in the first two years since release. But to get a picture of the risk you need to do more than count vulnerabilities. My full risk report was published today in Red Hat Magazine and reveals the state of security since the release of Red Hat Enterprise Linux 4 including metrics, key vulnerabilities, and the most common ways users were affected by security issues. It's all about transparency, highlighting the bad along with the good, and rather than just giving statistics and headlines you can game using carefully selected initial conditions we also make all our raw data available too so we can be held accountable.

Red Hat Enterprise Linux 5 was released back in March 2007 so let's take a quick look back over the first three months of security updates to the Server distribution:

This data is interesting to get a feel for the risk of running EL5, but isn't really useful for comparisons with other versions or distributions -- for example previous versions didn't include Firefox in a default Server installation.

As part of our security measurement work since March 2005 we've been tracking how the Red Hat Security Response Team first found out about each vulnerability we fix. This information is interesting as can show us which relationships matter the most, and identify trends in vulnerability disclosure. So for two years to March 2007 we get the following results (in percent):


I've separated the bars into two sections; the red sections are where we get notice of a security issue in advance of it being public (where we are told about the issue 'under embargo'). The grey sections are where we are reacting to issues that are already public.

The number of issues through researchers and co-ordination centers seem lower than perhaps expected, this is because in many cases the researcher will tell a group such as vendor-sec rather than each distributor separately, or the upstream project directly.

Red Hat has shipped products with randomization, stack protection, and other security mechanisms turned on by default since 2003. Vista recently shipped with similar protections and I read today an article about how the Microsoft Security Response Team were not treating Vista any differently when rating the severity of security issues. The Red Hat Security Response team use a similar guide for classification and I thought it would be worth clarifying how we handle this very situation.

We rate the impact of individual vulnerabilities on the same four point scale, designed to be an at-a-glance guide to how worried Red Hat is about each security issue. The scale takes into account the potential risk of a flaw based on a technical analysis of the exact flaw and it's type, but not the current threat level. Therefore the rating given to an issue will not change if an exploit or worm is later released for a flaw, or if one is available before release of a fix.

For the purpose of evaluating severities, our protection technologies fall into roughly three categories:

  1. Security innovations that completely block a particular type of security flaw. An example of this is Fortify Source. Given a particular vulnerability we can evaluate if the flaw would be caught at either compile time or run time and blocked. Because this is deterministic, we will adjust the security severity of an issue where we can prove it would not be exploitable.

  2. Security innovations that should block a particular security flaw from being remotely exploitable. Examples of this would be support for NX, Randomisation, and Stack canaries. Although these technologies can reduce the likelyhood of exploiting certain types of vulnerabilities, we don't take them into account and don't downgrade the security severity.

  3. Security innovations that try to contain an exploit for a vulnerability. I'm thinking here of SELinux. An attacker who can exploit a flaw in any of the remotely accessible daemons protected by a default SELinux policy will find themselves tightly constrained by that policy. We do not take SELinux into account when setting the security severity.

I've not been keeping a list of vulnerabilities that are deterministically blocked, but I have a couple of examples I recall where we did alter the severity:

Enterprise Linux 5 has been consuming much of my time over the last few months. From work on the signing server and new key policy, through testing of the new update mechanism, and continuing audits of outstanding vulnerabilities. Yesterday was release day, and we also pushed security updates for 12 packages in Enterprise Linux 5.

It may seem surprising that we release security updates for a product exactly at the same time we release it, but product development is frozen for some weeks before we release the product to give time testing from the various Quality Engineering teams as well as release engineering work. During that time we want to minimise the number of changes that will invalidate the overall testing, so we instead prepare the changes as updates. Since the vulnerabilities being fixed are already public, we push the updates out as soon as we can; holding them off to some scheduled monthly date would just increase customer risk.

Security advisories for Enterprise Linux 5 are available from the usual places, on the web, sent to the enterprise-watch-list mailing list, and via OVAL definitions. Red Hat Network subscribers can also get customized mails for the subset of issues that affect the packages they actually have installed.

For me, what's going to be interesting to watch over the next few months is how specific vulnerabilities and exploits affect this platform. Red Hat Enterprise Linux 5 packages are compiled both with Fortify Source and stack smashing protection in addition to all the security features that were in version 4. I'll be reporting on what difference this makes through the year.

We're changing the package signing key we use for all new Red Hat products.

Since 1999, all RPM packages in Red Hat products have been gpg signed by the master key "Red Hat, Inc <>" (keyid DB42A60E). I'll call this the legacy signing key for the rest of this article. This signature is one of two security mechanisms we use to ensure that customers can trust the installation of packages and their updates. The other is that the update client, up2date, checks the SSL server signature when it connects to the Red Hat Network to ensure that it only talks to official Red Hat servers; so removing the possibility of a man-in-the-middle attack.

From 2007, all new products will be signed with a different master key, "Red Hat, Inc. (release key) <>" (keyid 37017186). This includes Red Hat Enterprise Linux 5, and any other new products that use RPM packages. The exception to this rule is that any new layered products designed for older versions of Enterprise Linux will still use the legacy key: so for example, a new version of the Application Stack for Red Hat Enterprise Linux 4 will be signed with the legacy key.

The legacy key hasn't been compromised so why change keys? It's all to do with the way the keys are stored and managed internal to Red Hat. The legacy key is a software key and so the key material exists, protected by a passphrase, on a hard disk. When packages need to be signed one of the Red Hat authorised signers manually runs a signing command, this calls rpm --resign which asks for the passphrase then in turn calls out to GNUpg to do the actual signature creation. So the authorised signers not only had the ability to sign with the key, but they also have the ability to read the key material. In theory this means that a malicious internal signer could copy the key, take it away with them, and sign whatever and whenever they wanted. Or, more likely, a clever intruder who gained access to our internal network could perhaps capture the key and passphrase, compromising the key. The risks mean we've had to be really careful who has signing privileges with the legacy key and how the key signing is handled.

The new key, in contrast, was created in a hardware cryptographic device which does not allow the unprotected key material to be exported. This means we can give authorised signers the ability to sign with the key, but no one can ever can get access to the key material itself. This is an important distinction. If for example a current authorised signer switches roles and is no longer responsible for package signing we can instantly revoke their rights and know that they no longer have the ability to sign any more packages with that key.

There was no off-the-shelf solution available for hardware-based RPM key management, so we developed one internally ourselves. We used nCipher nShield hardware security modules (FIPS 140-2 validated) for the key protection along with custom patches I developed to interface RPM/GNUpg to the unit. At the same time we also introduced an extra layer of abstraction to the signing software, so we can authorize signers using their existing internal kerberos credentials.

So, as a customer, you won't really notice any difference. For Red Hat Enterprise Linux 5 you'll find the public keys on our website as well as in the /etc/pki/rpm-gpg/ directory and you'll be prompted when updating or installing new packages for the first time to import that new public key.

This change basically makes it easier for us to protect our signing key and reduce the risk of it being compromised, therefore reducing the chances we'll need to change the key and involve customer effort in the future.

<< prev [ 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 ] next >>

Hi! I'm Mark Cox. This blog gives my thoughts and opinions on my security work, open source, fedora, home automation, and other topics.