On OpenSSL versions in Solaris

(UPDATE 2009-09-10: see also 0.9.8k in OpenSolaris)

(UPDATE 2009-03-02: see also OpenSSL version string format changed on recent changes in the OpenSSL version string.)

Our customers ask frequently about OpenSSL version in Solaris 10, their usual question is "when is OpenSSL going to be upgraded to 0.9.8x in Solaris 10?". In this blog entry, I would like to clarify some FUD and explain what rationale is behind the fact that S10 is shipped with a version that is older than recent versions released by OpenSSL project and why that is not anything our security aware customers should be scared of.

Some background

Let's talk about some background first. OpenSSL is basically "just" a development toolkit implementing Secure Sockets Layer (SSL v2/v3) and Transport Layer Security (TLS v1), and is shipped together with a full-strength general purpose cryptography library. What I want to emphasize is that most people don't use OpenSSL directly. Why? It's simple - they don't need to. On the other hand, virtually all customers use OpenSSL indirectly through the use of Apache, SSH, Gnome and many other applications that make use of SSL2/3, TLS1 or generic crypto functionality shipped with OpenSSL.

There are definitely some customers who either make use of openssl(1) command line tool or who develop with OpenSSL libraries. Please see below what we suggest to do about that and I will continue on talking about the most common case where people use OpenSSL indirectly.

So, why to bother about different versions of OpenSSL in general? For two things. First is that every version add some new functionality. The second one is that every version may fix (and some definitely do) recently found security vulnerabilities.

Issue 1: is there any missing functionality in version shipped with S10?

Yes, there is, but as previously said missing functionality in a development toolkit is generaly not an issue provided that all the apps you or your customers use are doing fine which is usually the case. Few applications demand newly added functionality unless they really need it. That's the reason why all the tools/apps that use OpenSSL and are shipped with Solaris 10 just work. And if you download and build from sources it will mostly work fine, too.

Issue 2: is that version vulnerable?

Here is when the second topic comes up and what our customers are worried about - and they are right to be worried. They usually say: OpenSSL 0.9.7d shipped in S10, released in Mar 17, 2004 contains many security bugs! We need an upgrade! Well, there is no need for that and I'm gonna explain why. If you run openssl version on S10 box, you see:

$ openssl version
OpenSSL 0.9.7d 17 Mar 2004

...which is admittedly a little bit confusing. What it should read is something similar to what we can see in OpenSolaris:

$ openssl version
OpenSSL 0.9.8a 11 Oct 2005 (+ security patches to 2006-09-29)

The added note about security patches means that we fix every security vulnerability found in OpenSSL as soon as we can, both in OpenSolaris and Solaris 10 (and Solaris 9 where some part of OpenSSL lives, too). The fix is usually in OpenSolaris within a couple of days since it's released or since we get to know about it through background channels used to notify vendors shipping OpenSSL before the public disclosure of the vulnerability. We need those few days to prepare the fix, check it, properly test it to verify that it doesn't break something else, and put it back. In a couple of weeks there is a patch ready for S10 and S9. We are working on getting that "(+ security patches to ...)" string to the version output even in S10 so that you immediately know whether your OpenSSL version is vulnerable or not by comparing the date with the latest OpenSSL security advisory.

Why not to upgrade to the latest 0.9.7l in S10?

OK, there are differencies in functionality between current 0.9.7d and latest 0.9.7l and both versions are binary compatible (more on the topic later). So it seems that there is no reason why not to upgrade OpenSSL in S10 and get all the security fixes for free. However, there are good reasons why not to. One reason I already mentioned - most users don't need it at all so why to "fix" the version number.

Another reason is that it's not as easy as it looks. OpenSSL became an integral part of the system with many consumers relying on it. Let's name some at least: WAN boot, install software (yes, you cannot install Solaris without OpenSSL libraries any more), Apache, SSH and many others. Upgrading 0.9.7d to 0.9.7l would mean that we would have to spend literally a couple of months to be reasonably sure that nothing is going to break.

Of course, nothing is supposed to break but that's not good enough. No one can be reasonable sure without proper testing. One example for all. When upgrading from 0.9.7d to 0.9.8a in OpenSolaris, there was an undocumented change in behaviour of OpenSSL_add_all_algorithms(3). In 0.9.7d OpenSSL_add_all_algorithms(3) apparently did also what SSL_library_init(3) was supposed to do. According to the documentation, OpenSSL_add_all_algorithms(3) doesn't do anything about SSL stuff so that's fine. However, the problem was that some applications written some time ago relyied on that, using SSL without proper use of SSL_library_init(3). Without proper testing of everything we wouldn't have found out before shipping the new version. The project of upgrading OpenSSL in Nevada took roughly 5 months.

I would conlude that since there is no significant reason for a new OpenSSL version in S10 now, then, well, there is no need to upgrade. If you are interested why we don't upgrade to the newer 0.9.8x version in S10, read on. The answer for this question is also simple - it's quite difficult and definitely risky because of binary incompatibility between 0.9.7 and 0.9.8 branches. Let me explain why.

Binary incompatibility between different branches

Some basic facts first:

0.N.Mx (like 0.9.7d) version is binary compatible with 0.N.My (0.9.7i). If it's not then it's a bug and OpenSSL team will fix it very quickly.
0.N.Mx is NOT binary compatible with 0.N.Px or 0.N.Py. The incompatibility is a reason why M changed to P. Example is 0.9.7d in S10 and 0.9.8a in Nevada.
API compatibility is not an isssue
symbols in OpenSSL are not versioned but since DT_SONAME in a dynamic section of ELF dynamic object (= the OpenSSL library) contains all three numbers (0.N.M) then no application built against 0.N.M can be dynamically linked with 0.N.P by mistake. Linker just won't allow it.

Problem with multiple versions of the library in the system

Problem here can be if an application is indirectly dependent on more OpenSSL libraries with different DT_SONAME. Development, maintaining, building and cooperation with different teams developing different pieces of software can become very difficult. You can read about it for example from Debian people: bug 180725

So, that's the reason why wanted only one version of library in the system at any time when we initially integrated OpenSSL in the system back in 2003. I think it's a good idea considering the development process at Sun where many remote teams cooperate. Yes, it was spent substantial amount of time ensuring that all Nevada would transit smoothly from 0.9.7d to 0.9.8a but once done it works and will work. Some customers may not be happy but we could also ship 0.9.7, 0.9.8, then 0.9.9 when it's out etc. It would become a nightmare to maintain that. Some vendors ship both versions, some vendors only one at a time.

Note for BFU users in OpenSolaris: you may see both versions of OpenSSL in your system if you BFU'ed your system containing 0.9.7d with ON bits where OpenSSL had been already upgraded. Since apps from other consolidations like Intall commands, Apache, Gnome etc. were still dependent on older 0.9.7d then BFU couldn't have removed 0.9.7d. It would have crippled your system otherwise. However, next full system upgrade removes that old 0.9.7d from your system completely.

Why the ABI incompatibility at all? The problem is not that the community would change number of function call arguments or removed this call and added another one. The problem is that from historical reasons they pass structures to these calls and these structures are defined in public header files in the system include directory. And these structures must change from time to time as new things come in. No filter library choosing only a subset of calls or similar aproach is a solution. The answer from OpenSSL core developer Richard Levitte explains why hiding them now is a very difficult problem.

In short - ETOODIFFCLT. Note that OpenSSL 0.9.8a has around 270.000 lines of code.

Versioning of symbols might help with those indirect dependencies but that's not an issue for Solaris for now since we have just one version in the system.

Conclusion on ABI (in)compatibility

0.9.8x will be around for some time so the next upgrade in OpenSolaris won't break ABI compatibility. We will upgrade to the latest stable version before Solaris 11 goes out. And new applications must use EVP_ API only, not lower level functions to minimize potential problems with ABI incompatibility.

On a different note, the same developer Richard Levitte answering above started a project that should be a replacement for OpenSSL with all these problems fixed but this won't help us anytime soon. The project page is www.opentls.org.

If you really need some recent OpenSSL functionality in S10

...then please let us know and let us know why exactly you need it, your opinion is important to us. When we know what's happening out there we may do something about it.

Also I would like to suggest a workaround - you can use BlastWave project that ships up to date packages including OpenSSL; those packages are installed to a separate directory tree so no worries about overwriting your system. They are not supported by Sun of course, there is no support for PKCS#11 engine but installing them and using them might be an option for you.

Posted on: Oct 17, 2006

Posted by: janp

Category: Sun

Permanent link to this entry

Comments:

Thank you for explanation of "to/not to" upgrade openssl - saved me a Big headache of determining whether or not stuff would go bust! At least this has bought me a little bit more time to test out openssl 0.N.Py.

TY again!

Posted by z rao on May 28, 2008 at 07:29 PM CEST #

I need a openssl binary which will be used in libcurl.I tried in net to get that, but not able to get. I am getting only source code.
So could you please provide me that or let me know from where I can get it. My PC is solaris10-sparc.

regards,
Uttam

Posted by uttam on June 23, 2008 at 02:35 PM CEST #

Solaris 10 is shipped with OpenSSL by default. See /usr/sfw/lib directory.

Posted by Jan on June 29, 2008 at 08:35 PM CEST #

I tried to Compile Stunnel 4.25 with OpenSSL 0.9.7d on Solaris 10.

Configure:
# ./configure --prefix=/usr/local/stunnel-4.25

configure: **************************************** success

# Make

Make Error:
Undefined first referenced
symbol in file
BIO_set_flags protocol.o
SSL_CTX_set_info_callback ctx.o
ld: fatal: Symbol referencing errors. No output written to stunnel

What is the correct stunnel version for Openssl 0.9.7d in Solaris 10?

Posted by Nelgraine on September 22, 2008 at 07:43 AM CEST #

to Nelgraine: this looks to me like a building against 0.9.7x when 0.9.8x was used before, and not cleaned before the next build. Or building against a part of Stunnel installed before and built against 0.9.8x that time.

Stunnel 4.26 builds fine on my lab machine installed with s10u5. I haven't tried 4.25.

Posted by Jan Pechanec on September 22, 2008 at 09:20 AM CEST #

I havent notice that there is a default openssl 0.9.7d installed in my solaris 10 machine.
Ive installed openssl 0.9.8h without any errors or warning...

How to uninstall openssl 0.9.8h in solaris 10?

Posted by Nelgraine on September 24, 2008 at 03:01 AM CEST #

@Nelgraine: The only way to uninstall 0.9.8 is reverse the process used for installing it. So, if you typed 'make install' then the only option is to track the files and remove them by hand.

Posted by Vladimir Kotal on January 16, 2009 at 04:18 PM CET #

Is there someplace where I can find general dirctions on upgrading openSSL on a Solaris 10 x86 system? I've tried to upgrade to openSSL 0.9.8.j from the version that came with my server (OpenSSL 0.9.7d 17 Mar 2004 (+ security patches to 2006-09-29)) and when I run the openssl version command, it still shows 0.9.7d. I've searched and searched and cannot find too much on how to install this or upgrade it on a Solaris 10 x86 platform. Some instructions say to do the following: ./config...make...make all. Others say to gunzip and then pkgadd. So, I am not sure what to do.

Posted by William Estes on January 30, 2009 at 05:10 PM CET #

@William: I'd advice to upgrade OpenSSL in S10 only in case you very much know what you're doing. Installing new OpenSSL version in separate directory (say /opt/ or /usr/local/) and linking a specific application to the new libraries is one thing, replacing the OpenSSL libraries used by system is another (risky and totally unsupported) thing. For example, by replacing /lib/libcrypto.so you will loose the ability to leverage HW crypto (such as SCA6k card or built-in crypto accelerators in Niagara CPUs) by using PKCS#11 engine via EVP OpenSSL API.

Posted by Vladimir Kotal on February 12, 2009 at 10:33 AM CET #

Why is CVE-2008-5077 no being fixed for the OpenSolaris 2008.11 release? I see that a fix is contained in b107 and there is already a preliminary patch for Solaris 10.

Posted by Johann on March 02, 2009 at 05:57 PM CET #

@Johann: A Service Repository Update (SRU) for CR 6786120 is in the works. I putback (er, pushed) the fix to the OpenSolaris 2008.11 repository some time ago so it's definitely coming.

Posted by Vladimir Kotal on March 02, 2009 at 07:00 PM CET #

I'll give you a good reason. It's why linux is kicking solaris 10's butt. Admins and Developers want technology greater then 2006 when it's 2009. It's a joke that sun doesn't update their packages like samba, openssl just to name a few.

Posted by Mark on March 23, 2009 at 04:13 AM CET #

Mark, linux is not a system, it's a kernel. If you want to talk about systems, talk about linux distros. If you want 0.9.8 branch, you can use OpenSolaris (even with support). As of now, 0.9.8a is there, soon to be upgraded to 0.9.8j. You just cannot compare "linux" to Solaris 10.

and if you think that "linux" is superior to S10, use it. There are good reasons to keep the current version in S10 and use its next branch in OpenSolaris but I would be repeating myself.

Posted by Jan on March 23, 2009 at 09:43 AM CET #

Hello,
Currently using most current release 10/08. I am trying to use the cups on Sol 10 u4 companion disk. I get the following error.
ld.so.1: cupsd: fatal: libssl.so.0.9.8: open failed: No such file or directory
I have the blastwave openssl 0.9.8 installed and would like to use for this application. This wants to use 0.9.7.

Thank You,
David

Posted by David Hamilton on April 06, 2009 at 08:10 PM CEST #

David, the following says that "cupsd" really needs 0.9.8:

ld.so.1: cupsd: fatal: libssl.so.0.9.8: open failed: No such file or directory

check that you really have 0.9.8 from Blastwave installed. If you have the problem might be that cupsd(1m) doesn't have /opt/csw/lib in its search path. That could be because cupsd wasn't installed from blastwave or was built on another machine with OpenSSL 0.9.8 in a different directory.

Posted by Jan on April 07, 2009 at 10:04 AM CEST #

Jan, thanks for the entry, it's very enlightening, especially the linked updated post regarding the additional CVE's covered in the version string output.

However, there is still a significant time delta between the time an OpenSSL CVE is released and what I consider an appropriate patch from SUN. Case in point, CVE-2009-0590, CVE-2009-0789, and CVE-2009-0591.

These three could be summarized by the following:

CVE-2009-0590, CVE-2009-0789:
1) The vulnerability is due to an error in the ASN1_STRING_print_ex() function when printing BMPString or UniversalString strings. Remote attackers that can cause an OpenSSL-based application to print a BMPString or UniversalString can trigger a memory access flaw. Successful exploitation could result in a denial of service condition. (CVE-2009-0590)

2) The vulnerability is due to an error when processing malformed ASN1 structures. Remote attackers could exploit this vulnerability to trigger an access to invalid memory via a specially crafted certificate. Successful exploitation could result in a denial of service condition. (CVE-2009-0789)

CVE-2009-0591:
The vulnerability is due to an error that the CMS_verify() function incorrectly handles an error condition when processing malformed signed attributes. Remote attackers can exploit this vulnerability to trick an application into considering a malformed set of signed attributes valid and skip further checks. Successful exploitation requires access to a previously generated invalid signature and results in bypassing certain security restrictions.

All versions prior to 0.9.8k are vulnerable. I recognize CVE-2009-0789 was first identified on Win64 "and other" systems, but the OpenSSL project in general has since been added to the vulnerable list.

As you've pointed out, there are a large number of applications that use the OpenSSL libraries in Solaris 10. However, would you agree that the above CVE's represent a significant enough threat that it should trigger a very fast response from SUN and a patch within a few days that would address these CVE's?

I don't deny that having the CVE's included in the version string output is great. I think addressing the CVE's faster would be better. Is there any chance this time delta will be shorter in the future, and in your opinion, what would be the best way to get SUN to issue a security patch for those three outstanding OpenSSL CVE's?

Posted by Darren Price on April 14, 2009 at 04:47 PM CEST #

to Darren: from those 3 bugs only CVE-2009-0590 is relevant to Solaris (other one doesn't affect our architectures, and the last one was introduced in a later version than we have). The fix was already put to Nevada, and patches for S10 should be released soon. I'm not in the team responsible for releasing patches for S10 but what I understand is that you can have preliminary release of the patch if you have the support contract. If you have that please use your usual contact.

BTW, 0.9.8k should appear in Nevada hopefully within a few weeks and it should be easier to update it in timely manner then. However, S10 most probably stay with 0.9.7.

Posted by Jan on April 14, 2009 at 05:12 PM CEST #

Great Jan, thanks for the response & additional information. We do have contract support so I'll investigate that early-release option.

Posted by Darren Price on April 14, 2009 at 05:20 PM CEST #

Hi
I am using the Solaris 10 64bit sparc /usr/sfw/lib/sparcv9/libcrypto.so.0.9.7 and libssl libraries to implement a TLS connection and have a TLS handshake failure which I believe to be due to a -xdepend optimisation bug in the Sun Studio 11 compiler, see

6353992 openssl-0.9.7i test fails in run-time under -xO3 -xdepend
and
http://rt.openssl.org/Ticket/History.html?id=1281

Is there a patch available which includes these libraries compiled with either a patched Sun Studio 11 or Sun Studio 12 compiler?

P.S.
I have tried the libcrypto and libssl libraries in patches 121229, 139555 and 141742 without success.
I have also built a local version of OpenSSL 0.9.7d with -xdepend=no on Sun Studio 11 and one with the default solaris64-sparcv9-cc flags on Sun Studio 12 both of which work. However, I would prefer not to ship locally built openssl libraries with my application.

Posted by Ron on June 25, 2009 at 12:16 PM CEST #

@Ron: Sun Studio 11 is not used for compling Solaris 10 update releases so the cause is probably different (assuming the bug was not present in earlier Sun Studio releases). Please raise a support call if this problem is reproducible.

Posted by Vladimir Kotal on June 25, 2009 at 02:12 PM CEST #

"The fix is usually in OpenSolaris within a couple of days since it's released or since we get to know about it through background channels used to notify vendors shipping OpenSSL before the public disclosure of the vulnerability. We need those few days to prepare the fix, check it, properly test it to verify that it doesn't break something else, and put it back. In a couple of weeks there is a patch ready for S10"

I find this not to be the case for S10. For example the latest updated for openssl is buried in a patch for sshd. And then this fix is only for CVE CVE-2009-0590.

There are two outstanding vulnerabities that require to be addressed.

CVE-2009-0591
CVE-2009-0789

All of these have been address by openssl in there version k dataed March 2009.

When will these fixes be available for S10?

Posted by Davsanto on August 04, 2009 at 07:09 PM CEST #

The UNIX STIG now requires OpenSSL 0.9.8k ...
Government programs are audited using SRR scripts and checklists.
Programs fail audits with 0.9.7d installed with Sol 10 5/09. even with patches. This may be a situation where Sun would want to stay ahead in this market with customers and update OpenSSL and OpenSSH more frequently. When is this upgrade expected?

Posted by Abrams on August 04, 2009 at 11:57 PM CEST #

to Davsanto: CVE-2009-0591 does not affect the OpenSSL version in S10 since only 0.9.8h or later is affected, CVE-2009-0789 does not affect Solaris as such (see the vulnerability report if interested why is that).

Posted by Jan on August 07, 2009 at 01:12 PM CEST #

to Abrams: thanks for the information, it's very good to know that.

anyway, if UNIX STIG complains about security issues of a fixed OpenSSL implementation then there is something wrong about the tool, I would say...

I really think that security tools that don't test the problem but toy with version numbers only are of limited use. What if someone forks the project, rename it, use a different numbering scheme, and NOT fix the vulnerability. The security tool will be probably happy about that. I agree that some people will be influenced by the (wrong) STIG's output but I explained above why we are most certainly not going to upgrade OpenSSL in S10 to 0.9.8x. There are technical reasons for that.

also, please note that we don't use OpenSSH, we use SunSSH which is a fork of OpenSSH about 6 years ago. Please see this for more information:

http://www.opensolaris.org/os/community/security/projects/SSH/

Posted by Jan Pechanec on August 07, 2009 at 01:25 PM CEST #

Jan,

Concerning CVE-2009-0789. If you mean the vulnerability report found here (http://www.openssl.org/news/secadv_20090325.txt) to me it reads that bad certificates can be created on only certain systems but that those bad certs will affect all systems who read them.

Posted by Greg on August 10, 2009 at 10:07 PM CEST #

to Greg: nope, it means that if you are on the platform vulnerable, any application using OpenSSL, whether it's a client, server, or anything else, is vulnerable, not just an SSL server, for example.

Posted by Jan on August 12, 2009 at 09:33 PM CEST #

Jan,

Besides patching the holes identified in the CVE's, you also need
to work with scanning software vendors to make sure that they don't
identify you as vulnerable. As it is now, we get hit with reports
from Security complaining about our out-of-date version of openssl.
The guys running the scanners are too busy to deal with written
exceptions. It is much better for everyone if it just doesn't get
flagged in the first place.

Posted by John on September 15, 2009 at 01:29 AM CEST #

hi John, I assume you assume we are not busy here at all :-) Seriously, as I said before, scanners that just compare strings are not very helpful, in my opinion. A lot of applications are borrowing code, possibly buggy, from other open source projects and since those scanners work with version strings instead of the real vulnerabilities, they might be missing real problems and report false positives.

Anyway, this problem is in S10 only now since we plan in OpenSolaris to follow closely new OpenSSL version releases, and I think that those vendors could take into account that Solaris 10 exists :-).

Posted by Jan on September 15, 2009 at 10:05 AM CEST #

I'm a sys admin for the DoD and like others who have commented, I am obligated to maintain Solaris systems according to strict mandates of the UNIX STIG and TCNO's which identify specific CVEs.

I am specifically concerned with 2 openssh CVE's:
1) CVE-2009-1377
2) CVE-2009-1378

Nowhere do I find any discussion of the status of Solaris 10 openssh/sunssh. I have looked all over sunsolve and through sunalerts -- nothing. While I do not see solaris 10 labelled "vulnerable" at sites such as securityfocus, I also do not find solaris 10 listed under "not vulnerable".

Both of these CVE's date from May 2009.

Running "openssl version" on one of my solaris 10 boxen yields this:
OpenSSL 0.9.7d 17 Mar 2004 (+security patches to 2006-09-29)

Obviously, patched to 29 Oct. 2006 does would not include fixes in response to CVE's raised in May 2009.

I would like Sun to respond specifically to the vulnerabilities as they arise either say "not vulnerable" or "fixed in v. XXX"

Many thanks.

Posted by John@WPAFB on September 29, 2009 at 08:36 PM CEST #

John, please use your usual support channel to get specific information on S10. Anyway, both CVE reports say that only 0.9.8* are affected so S10 is fine.

Also, from your openssl version output I can see that you are not running the latest patches.

Posted by Jan on September 30, 2009 at 10:28 AM CEST #

Jan,

I think John's (and my) frustation is due to the fact that Sun doesn't seem to address CVE's unless an OS is specifically vulnerable to them. Some of the other vendors make statements, much like the following on CVE-2009-1377(XXXXXX being substitued for the vendor:

"Vendor Statements (disclaimer)
Official Statement from XXXXXX(09/02/2009)
This issue did not affect versions of openssl as shipped in XXXXXX Linux 3 and 4. This issue was addressed for XXXXXX Linux 5. Note that both the DTLS specification and OpenSSLs implementation is still in development and unlikely to be used in production environments. There is no component shipped in XXXXXX Linux 5 using OpenSSLs DTLS implementation, except for OpenSSLs testing command line client - openssl"

We are required to provide justifications as to why we aren't applying patches. So, unless a vendor releases a patch, we have to "assume" the OS isn't vulnerable, but that isn't a valid justification in most cases.

It would be very helpful for us if Sun were to provide a short statement on CVE's that Solaris is NOT vulnerable to, and why.

Who would you recommmend we contact to see if Sun could do this in the future? This would simplify things significantly for us.

Bill Rushlow

Posted by William Rushlow on October 01, 2009 at 06:20 PM CEST #

to Bill: I'm afraid that if the security team starts releasing statements on CVEs that Solaris is not vulnerable to, it would be a precedent for other SW as well. I'm afraid this is not going to happen.

Anyway, I assume you have support for Solaris, please use your usual channel to contact Sun about this. Unfortunately I don't know who you should talk to about this.

Posted by Jan on November 10, 2009 at 01:55 PM CET #

Jan Pechanec's weblog