PSMail and Heartbleed

In the wake of April 7th’s disclosure of CVE-2014-0160, PSMail's team has been working - along with a number of other service providers and software vendors - to fix this issue. Though PSMail's servers use a number of OpenSSL libraries that were found vulnerable, we have taken this opportunity not only to make the necessary changes as quickly as possible, but to step back and analyze the broader issue, to identify what areas might be affected, and to work to provide the most comprehensive fix.


Summary:
Our analysis of network and log data from PSMail servers show that this vulnerability is unlikely to have been used to reap PSMail user's private data (passwords, emails, etc.) from our servers.  Using a search of all possible indicators (network events, system logs and security events), we looked all possible outcomes of such an attack (much like the Monte-Carlo simulation with deterministic outcomes).  The analysis is simple yet effective in analyzing attempts for exploiting such vulnerability on our server.  Attack scenarios and analysis is detailed below:

1. Stealing of private keys and certificates:

Using a penetration testing analysis of various SSL reaping scripts and simulating possible outcomes/indicators in logs, IDS events, and system alerts, we have determined that it is highly unlikely that any of our certs were stolen. We know this from the fact that these types of reaping run thousands, if not millions, of queries against a server, and such activity has not been seen in the past 3 months of the logs and IDS alerts we maintain on our server. These logs and alerts include studying a single network resource or a single behavior (bytes/packet) from multiple network systems to look for one that resembles a harvest of private information on our server.  Read related ...

2. Attack access methods

The attack is most effective when using HTTPS and reaping information from servers that perform authentication. This would be our primary mail system, https://mail.psmail.net/, and our VPN server, https://m.psmail.net.

Our VPN server was patched against such an attack about 3 months before the vulnerability was released. This was basically because the updated image on our VPN server just does not allow the heartbeat feature of SSL to be accessed (heartbeat feature is what enables this exploit). This was coincidental and not planned specifically to mitigate this vulnerability.

On our mail server, an attempt to reap any information is restricted by a number of factors. While OpenSSL is vulnerable, we also protect our mail servers with memory protection by running micro processes and restart of services to clear much of the memory cache of sensitive information. This makes it difficult to gather past information from HTTPS sessions. Analysis of our logs from the past 6 months reveals no attempt, either from a single network address or from a behavior (successful scan) that will allow someone to reap large amounts of data over the network from PSMail servers.

Our other servers that are exposed over the Internet do not carry any user or private data except that which is publicly available through secure channels. These servers, such as https://info.psmail.net/, do not carry any session information or user's data.

3.   Patch cycle and our security monitoring / auditing

Our mail servers were patched about 12 hours after the public release of the vulnerability (April 7th, 2014). This is a reasonable response time. Our VPN servers had already been updated 3 months before, disabling this attack access method. Analysis of logs during our 3 month log retention only shows number of scans after vulnerability exposure from a small amount of network addresses using many public services such as Symantec's heartbleed check, Qualys SSLlabs, http://filippo.io/Heartbleed home-grown heartbleed test, and similar ones against our mail servers. Similar attempts on our public server such as www.psmail.net and info.psmail.net were not analyzed as they don't carry any sensitive information.

4.  Web session protection and session hashing

 Our critical server, mail.psmail.net, has a further HTTPS session protection. Any private information, such as a stolen session key or HTTP Cookie, cannot be reused on our server. If a session is broken with either a network change (network address, traffic hops) or through an access method change (browser, concurrent session) we completely destroy the temporary session forcing the user to login once again.  While this seems inconvenient, we have always felt that this is a critical way to protect even logged in sessions that are provided via our mail system.

Note: Heartbleed, although simply explained (in blogs, news, etc.) and frequently discussed on the web, is a complex attack vector. This can be confusing for non-technical users as there are several factors in play to successfully complete such an attack. Strangely, these include factors that may not have been considered, such as how busy the SSL offloading servers/systems are, how many of these servers exist, and what private information (if any) reside in them. We recommend that you consult experts directly to help you better understand the issue. If you are able to simulate this in a test environment, many of these factors will become as obvious to you as they have been to our technical team.

At PSMail we take security very seriously and continue to do our best to respond to these types of emergencies in a calculated way. Although it is unlikely that any subscriber data was compromised, we recommend that you continue best security practices, such as changing your passwords regularly, using passwords only on systems that you own and not from public computers, and adopting reliable web password access methods like those we already provide (i.e. Software Token based passwords that require a pin number and a randomly generated password code for each web login).