PDA

View Full Version : HOWTO: Upgrading OpenSSL


The Prohacker
06-28-2003, 01:42 AM
Title: Upgrading OpenSSL on Redhat Servers
Revision: 1.0.5
Date: June 28th, 2003
Time: 2:50am CST
Description: Guide for installing and upgrading OpenSSL services on a typical Redhat server to close the very common OpenSSL hole that is plaguing many servers today.
Difficulty: 6/10

------------------------------------------------------------

Disclaimer:
THIS HOWTO IS PROVIDED BY NULL-ROUTE DOT COM "AS IS" AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL NULL-ROUTE DOT COM. OR ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

------------------------------------------------------------

Introduction:
Many servers today even have a common hole that allows anyone local access to a server as the apache user. The main part of the exploit was found to be in mod_ssl 2.8.12 but many underlying faults lay in OpenSSL installation and libraries bellow version 0.9.6j. Upgrading mod_ssl to version 2.8.14 generally slows an intruder down but will not stop someone who is determined if they know about the older OpenSSL libaries.

This howto will explain how to properly remove older symlinks made by the Redhat installer and give details on the installation of a secure version of OpenSSL.

NOTE: This guide is for installing OpenSSL 0.9.6j. At the time of this writing I felt the 0.9.7x series was still to new for a production environment. But using 0.9.7x should work fine in all instances.

After upgrading the OpenSSL binary and libraries any other program that depends on them will also need to be recompiled to take advantage of the new libraries unless they were compiled with dynamic library support (most applications are compiled statically).

This howto assumes the reader has a general knowledge of Linux and the command line interface for installing applications and has root access to the server. Readers with minimal knowledge of Linux or the processes of compiling and installing software should not attempt this without proper supervision from a system administrator or in a production environment.

Pre-Installation:
Before beginning installation you should verify that your server is indeed open to this exploit because of an older OpenSSL installation. To verify at command line type:

openssl version

It should return something like this:


[root@grep root]# openssl version
OpenSSL 0.9.6b [engine] 9 Jul 2001
[root@grep root]#


If your version is bellow 0.9.6j. Then you are most likely open to these exploits and should continue on with this upgrade. If you show version 0.9.6j there is still a very good chance that you are open to this exploit and you must run another check to see if Apache has been compiled with an older version of OpenSSL.

To run this check visit this script: http://null-route.com/httpv.php
Note: This script is privately hosted by myself and will not log any server addresses you enter into it.

After running the second check you should get a result looking like this:


Server domain.com: Apache/1.3.27 (Unix) mod_ssl/2.8.14 OpenSSL/0.9.6b PHP/4.3.2 mod_perl/1.27 FrontPage/5.0.2.2510


As you can see the server is running OpenSSL 0.9.6b and would be open to many exploits even though mod_ssl is running the latest version.

Installation and upgrading:
To begin installation we will need to download OpenSSL 0.9.6j source tar ball to the server we will be upgrading. I generally put all my source files in /usr/src to keep them for later incase I need them.


# cd /usr/src
# wget http://www.openssl.org/source/openssl-0.9.6j.tar.gz
# tar -xvzf openssl-0.9.6j.tar.gz
# cd openssl-0.9.6j/


You should now be in /usr/src/openssl-0.9.6j/

We need to now remove all old symlinks to older version of OpenSSL before we begin.


# rm /usr/lib/libssl.so.0
# rm /usr/lib/libssl.so
# rm /lib/libssl.so.2
# rm /usr/lib/libcrypto.so.0


After removing all old symlinks to old versions of OpenSSL we can begin configuring and compiling the new version.
NOTE: On older systems this may take several hours. For a P4 2.4Ghz equiped with 1gb RAM it took me around 20 minutes to fully configure/compile/install OpenSSL. Your build times will vary depending on equipment and current load of the server.

We now begin by configuring our new installation of OpenSSL with shared libary support and threading support turned off to increase speed with mod_ssl.


# ./config no-threads shared


After config has ran we will now make/compile our build of OpenSSL. We will use one command, make, but will run it 3 times with differnt operators. Run these commands in this order. These will take some time:


# make
# make test
# make install


NOTE: make test requires BC math functions to be installed on your server to test the OpenSSL install.

After make install has been ran we will now export our libary path for OpenSSL:


# echo "/usr/local/ssl/lib" >> /etc/ld.so.conf
# ldconfig -v


We also need to recreate the symlink /lib/libssl.so.2 and /usr/lib/libcrypto.so.0 to our new OpenSSL libary.


# ln -s /usr/lib/libssl.so.0.9.6 /lib/libssl.so.2
# ln -s /usr/lib/libcrypto.so.0.9.6 /usr/lib/libcrypto.so.0


At this point you are done upgrading OpenSSL but your system isn't secure. You will need to recompile the other applications that used the old OpenSSL libaries at build time to fully seucre your system. Applications such as OpenSSH and Apache+mod_ssl are the ones you will absolutely need to recompile.

NOTE: I will try to do a howto covering the upgrading of OpenSSH within a few days to aid in the proccess.

NOTE: If after buiding OpenSSL openssl version still reports an older version you will need to run make clean and add --prefix=/usr to the config command I used earlier..


# ./config --prefix=/usr no-threads shared
# make
# make install


Closing:
If you have found any glaring typos, or outdated info in this document, please let me know.
The Prohacker, prohacker[at]null-route.com

exposed
08-05-2003, 11:18 AM
Upgrading was gaving some problem with some older software i have and now i want back to the default openssl rpm from redhat.

Installing the rpm with rpm -ihv goes ok but when i do openssl version i still see's the newest version i updated manual (tar.gz) any idea how i can return to the basic redhat openssl rpm again?

The Prohacker
08-05-2003, 01:01 PM
To go back you need to install the main openssl rpm and the deval rpm....

I beleive for RH 7.3 and 8:
openssl-0.9.6b-32.7.i386.rpm
openssl-devel-0.9.6b-32.7.i386.rpm

thoroughfare
10-31-2003, 10:43 AM
Hi,

I tried upgrading OpenSSL using the latest version (http://www.openssl.org/source/openssl-0.9.7c.tar.gz) but I've ran into problems.

If I do:

locate libsssl.so.0.9.7c

then it returns nothing, and when I do:

locate libssl

the only thing that looks similar to the file 'libssl.so.0.9.7c' from:
# ln -s /usr/lib/libssl.so.0.9.6 /lib/libssl.so.2

is 'libssl.so.0.9.6b'.

Shouldn't there exist a file 'libssl.so.0.9.7c'?

I can't restart the httpd service because of this, and nothing that relies on libssl is working.

I'd gratefully appreciate any advice.

Many thanks,
Matt

thoroughfare
10-31-2003, 05:17 PM
I figured it out in the end, just followed the instructions here:

http://marc.theaimsgroup.com/?l=openssl-users&m=103243136521648&w=2

and it worked out.

Thanks :)
Matt

thoroughfare
10-31-2003, 05:24 PM
Actually no, it's httpd still won't restart... and when I tested wget, I got this:

[root@xxxxxxxxxx customapache]# wget http://files.directadmin.com/services/customapache/build
--19:22:23-- http://files.directadmin.com/services/customapache/build
=> `build.2'
Segmentation fault


Help! :)

Matt

thoroughfare
10-31-2003, 05:33 PM
Some progress... when I run

[root@xxxxx logs]# httpd

I get this:


Syntax error on line 1506 of /etc/httpd/conf/httpd.conf:
Invalid command 'SSLEngine', perhaps mis-spelled or defined by a module not included in the server configuration


Is this because I need to update the OpenSSL engine too?

cheers,
M

ProWebUK
10-31-2003, 05:41 PM
Try running the upgrade again. It seems there was a problem during the upgrade. If this fails i suggest you use another copy of OpenSSL, still stable and secure but hopefully wont give the problem.

Chris

thoroughfare
10-31-2003, 05:54 PM
Thanks Chris, sorry for being a pain to everyone :)

Matt

thoroughfare
10-31-2003, 06:38 PM
Fixed it... kinda :)

I used openssl-0.9.6k.tar.gz instead of 0.9.7c, followed the install instructions, and httpd works now :)

Although I may be completely wrong, I think the problem was to do with this: http://www.openssl.org/support/faq.html#MISC6

...version 0.9.7c doesn't include a seperate OpenSSL engine, so it probably didn't exist when httpd was looking for it.

On the other hand, if I type 'httpd' at the terminal just by itself, I still the error:

Syntax error on line 1506 of /etc/httpd/conf/httpd.conf:
Invalid command 'SSLEngine', perhaps mis-spelled or defined by a module not included in the server configuration


Still not sure what's causing this but at least httpd is happy now :)

jasonyates
11-01-2003, 05:17 AM
Hey,

I get that error as well:


Syntax error on line 1508 of /etc/httpd/conf/httpd.conf:
Invalid command 'SSLEngine', perhaps mis-spelled or defined by a module not incl
uded in the server configuration


Jason

ProWebUK
11-01-2003, 06:05 AM
I think i know your problems....


the error line should be something like:

LoadModule ssl_module /lib/libssl.so.X.X.Xa

change the X.X.Xa

with the version stated on the compiled version, eg:

OpenSSL 0.9.7c

I recommend you backup yout httpd.conf before making changes firstly also.

Chris

thoroughfare
11-01-2003, 06:58 AM
Hi,

The line in httpd.conf said:

LoadModule ssl_module modules/libssl.so

So I located all instances of files called libssl.so, removed the most of them and replaced them with symlinks, linking to the /usr/local/ssl/lib/libssl.so file which is linked to the actual libssl.0.9.6 file in the same directory. So now all instances of libssl.so are symlinks apart from the actual file itself.

I'm still getting the error though, and now httpd doesn't restart again :)

Good job this isn't a production server :)

Matt

thoroughfare
11-01-2003, 07:04 AM
OK, I found two instances of 'SSLEngine' in httpd.conf, one in the general section of the file towards the beginning, and the second was here:

<VirtualHost 123.12.12.123:443>
ServerName localhost
ServerAdmin webmaster@xxxxxxx.xxx
DocumentRoot /var/www/html
ScriptAlias /cgi-bin/ /var/www/cgi-bin/

SSLEngine on
SSLCertificateFile /etc/httpd/conf/ssl.crt/server.crt
SSLCertificateKeyFile /etc/httpd/conf/ssl.key/server.key
</VirtualHost>


And it seemed it's been these lines that are causing the problem, so commented out just the last 3 (SSLEngine, SSLCertificate*) and now httpd is restarting fine.

I left the first instance of SSLEngine uncommented. So why is one instance causing problems but not the other?

Cheers,
Mattt

ProWebUK
11-01-2003, 07:27 AM
you did make backups of the files you removed now.... didnt you? :D

thoroughfare
11-01-2003, 08:03 AM
Erm, nope... didn't think I'd need to... I mean they're all supposed to be the same file, so I just pointed them all at the actual file.

I've got a backup image of my filesystem if I need it.

Plus, I can always just reinstall OpenSSL right?

Sorry if I seem super-dumb at all this, I've been working with PCs since the age of 8 but I've only touched linux in the last year (got RH 8 on my 2nd HDD) and I've never adminned a server before :D Gotta start somewhere tho :D (Before a few days ago, I didn't even know how to use vi, or any other Linux command for that matter.) :D

Matt

ProWebUK
11-01-2003, 08:35 AM
ok possibly try the latest openSSL or recompile the one you just installed, if you have problems restarting apache check the error location and the line, take a look in the file, go the correct line (press ctrl + c to check your current line - in pico) and check what i stated above........ also check the any paths on the specified line or just above exist.

Chris

thoroughfare
11-01-2003, 09:36 AM
Line 1506 (the problem line) is just:

SSLEngine On

Cheers,
Matt

DirectAdmin Support
11-01-2003, 02:08 PM
Hello,

Remember to use:

service httpd restart

and not the httpd binary directly.

John

thoroughfare
11-01-2003, 02:44 PM
Hi,

I have been using service httpd restart; I used httpd before because it proved helpful in finding out what was stopping httpd from restarting when service httpd restart wasn't outputting any errors but was still failing.

I've been researching my problem and it seems there are others who've experienced the same, so I'm going to work on it for a while and post later with a fix hopefully ;)

Thanks for everyone's help so far...

Matt :D

thoroughfare
11-01-2003, 03:57 PM
Still no luck. I've tried upgrading mod-ssl to the latest version for Apache 1.3.28, and recompiled apache and openSSL, still no luck.

And I'm using service httpd restart.

After doing some Googling, I read somewhere I should try commenting out the IfDefine SSL tags around the SSL configuration options and the LoadModule mod_ssl line, but after doing that I only got:

Can't locate API module structure `ssl_module' and undefined symbol: ssl_module.

Anybody?

Much appreciated,
Matt

bvvelzen
11-06-2003, 08:11 AM
I have exactly the same problem.

When I trie to start apache bij service httpd start (becase it's not running), I get this messages:

/etc/init.d/httpd: line 46: ulimit: open files: cannot modify limit: Invalid argument
/etc/init.d/httpd: line 47: ulimit: open files: cannot modify limit: Invalid argument
/etc/init.d/httpd: line 48: ulimit: open files: cannot modify limit: Invalid argument
/etc/init.d/httpd: line 49: ulimit: open files: cannot modify limit: Invalid argument
Starting httpd: Syntax error on line 34 of /usr/local/directadmin/data/users/cduyzer/httpd.conf:
Invalid command 'SSLEngine', perhaps mis-spelled or defined by a module not included in the server configuration


And I also read that I must add this line:
LoadModule ssl_module /lib/libssl.so.0.9.7a

to /etc/httpd/conf/httpd.conf

but when I then started apache I get this messages:

/etc/init.d/httpd: line 46: ulimit: open files: cannot modify limit: Invalid argument
/etc/init.d/httpd: line 47: ulimit: open files: cannot modify limit: Invalid argument
/etc/init.d/httpd: line 48: ulimit: open files: cannot modify limit: Invalid argument
/etc/init.d/httpd: line 49: ulimit: open files: cannot modify limit: Invalid argument
Starting httpd: Syntax error on line 243 of /etc/httpd/conf/httpd.conf:
Can't locate API module structure `ssl_module' in file /lib/libssl.so.0.9.7a: /usr/sbin/httpd: undefined symbol: ssl_module

I have this openssl version:
OpenSSL 0.9.7a Feb 19 2003

What's wrong?

thoroughfare
11-06-2003, 09:56 AM
I got:

Can't locate API module structure `ssl_module' in file /lib/libssl.so.0.9.7a: /usr/sbin/httpd: undefined symbol: ssl_module

also, but didn't post because I got sick in the end and just started with a clean server again; I've yet to upgrade OpenSSL because of this.

Cheers,
Matt :)

The Prohacker
11-06-2003, 11:00 AM
Did everyone recompile mod_ssl?

thoroughfare
11-06-2003, 11:12 AM
I'll try it again in a few hours and try recompiling mod_ssl.

Cheers,
Matt

synergy
11-11-2003, 02:43 AM
I had similar problem to those here. I followed the note at the beginning of the thread (which I missed the first few times) changing:


# ./config no-threads shared

to

# ./config --prefix=/usr no-threads shared


the update of openssl seemed to work fine.

MagnuM
01-13-2004, 04:55 AM
OK so I have the same problem with this error (when running apachectl configtest):
Invalid command 'SSLEngine', perhaps mis-spelled or defined by a module not included in the server configuration

The point is that I am not so advanced in compiling and installing Linux software.

I try this: [b]locate libssl.so[/], and I do get

/usr/lib/apache/libssl.so
/usr/local/directadmin/customapache/apache_1.3.29/src/modules/ssl/libssl.so
/usr/local/ssl/lib/libssl.so.0.9.7
/usr/local/ssl/lib/libssl.so.0
/usr/local/ssl/lib/libssl.so
/usr/src/openssl-0.9.7c/libssl.so.0.9.7
/usr/src/openssl-0.9.7c/libssl.so.0
/usr/src/openssl-0.9.7c/libssl.so
/lib/libssl.so.4
/lib/libssl.so.0.9.7a
/lib/libssl.so.2


So I found that I have an OpenSSL 0.9.7c archive and a directory in /usr/src, but I am not sure that this version is installed, because if I type openssl version I get 0.9.7.a as an answer.

Maybe I need to reinstall OpenSSL and recompile mod_ssl, but I afraid not to brake things. About the OpenSSL upgrading I read the intructions in this post, but don't know how to recompile mod_ssl.

DirectAdmin Support
01-13-2004, 10:28 AM
Hello

1) apache restart:
service httpd restart

2) rebuild mod_ssl
cd /usr/local/directadmin/customapache
./build clean
./build apahce_mod_ssl

John

MagnuM
01-14-2004, 01:35 AM
Thanks,

but when I try ./build apache_mod_ssl, it started the make procedure, but receive an error like this:

===> src/modules/frontpage
/bin/sh: line 1: cd: frontpage: No such file or directory
make[3]: *** [all] Error 1
make[2]: *** [subdirs] Error 1
make[2]: Leaving directory `/usr/local/directadmin/customapache/apache_1.3.29/src'
make[1]: *** [build-std] Error 2
make[1]: Leaving directory `/usr/local/directadmin/customapache/apache_1.3.29'
make: *** [build] Error 2

*** The make has failed, do you want to try to make again? (y,n):


Well, trust me I am sorry that make you guys mad at me, but I am starting to get familiar with Linux style software installation.
Anyway I want to upgrade to openssl-0.9.7c and OpenSSH 3.7, but I read that OpenSSL upgrade wil require to recompile any software which uses it (OpenSSH and Apache+mod_ssl).

So before trying to upgrade OpenSSL I was trying to recompile Apache+mod_ssl, just to see if this upgrade works fine, but it happens to brake.

Can you give me more suggestions, of how should I do that.

DirectAdmin Support
01-14-2004, 10:58 AM
Hello,

Re: Apache Breaks

cd /usr/local/directadmin/customapache
rm -f configure.*
./build clean
./build update
./build allJohn

MagnuM
01-15-2004, 01:02 AM
OK, thanks :)

The question is:
If I upgrade OpenSSL, do I need to recompile OpenSSH?

Because if I need to do that, I would like to upgrade to OpenSSH version 3.7, because I am running version 3.5. I also read the HOWTO, about upgrading OpenSSH, but I have experiencing some problems with PAM: http://www.directadmin.com/forum/showthread.php?s=&postid=9852#post9852.

So the main steps should be in that order:
1. Upgrade to openssl-0.9.7c
2. Recompile apache+mod_ssl
3. Upgrade to OpenSSH 3.7

Do I miss somehting, I mean do I need to recompile another software, beside those?

MagnuM
01-27-2004, 05:56 AM
I just follow the instructions from here http://marc.theaimsgroup.com/?l=openssl-users&m=103243136521648&w=2, and upgrade to openssl 0.9.6l.

The point is that I am still receiving this:
Syntax error on line 1508 of /etc/httpd/conf/httpd.conf:
Invalid command 'SSLEngine', perhaps mis-spelled or defined by a module not included in the server configuration

If I type openssl_version I receive: OpenSSL 0.9.6l 04 Nov 2003, and not OpenSSL 0.9.6l [engine] 04 Nov 2003, as I think it should appear. So the [engine] string is missing.

I have installed the folowing packages:
openssl-0.9.6l.tar.gz
openssl-engine-0.9.6l.tar.gz

Is it possible that the SSLEngine is not installed correctly?
What should I do, because my Apache is failing to start?

DirectAdmin Support
01-27-2004, 09:52 AM
Hello,

don't use apachectl, use:

service httpd restart

John

MagnuM
01-27-2004, 02:37 PM
I always use service httpd restart, because you write that on this forum many times, and I know it. Anyway I recompile Apache, and now it is working good.

I just don't know if the SSLEngine is installed correctly?

RTKS
02-24-2004, 12:51 PM
It seems this one is a bit more complicated.

1. Is this a really essential patch?
2. What is the security hole?
3. What would I need to recompile after getting the new version of SSL installed?

ProWebUK
02-24-2004, 02:06 PM
Originally posted by RTKS
1. Is this a really essential patch?
2. What is the security hole?
3. What would I need to recompile after getting the new version of SSL installed?

1) no, it's not "essential" - your server wont start steaming due to the fact its installed and out of date, however if the upgrade is to patch a security hole it should be upgraded - if its difficult its difficult although would you prefer spend some time and be safe or just say oh well, it looks difficult lets leave ourselves wide open and vulnerable to an attacker that knows the hole.

The guide was intended a while back for a 0.9.6j > 0.9.7* upgrade... since then there have been many more bug and security fixes, the latest version is 0.9.8 (after 0.9.7c)

2) You can view ALL changes / fixes at http://www.openssl.org/news/changelog.html

3) Most things should be ok, you may have to change a few paths...

Chris

RTKS
02-24-2004, 02:27 PM
Should the guide still work substituting the new version for old?

ProWebUK
02-24-2004, 02:31 PM
yes.

Chris

RTKS
02-24-2004, 04:36 PM
I think this is one of those times I need to step away from the keyboard. I can't get the version to update from .6b for the life of me despite installing .7c.

Any ideas at all?

vandal
05-06-2004, 06:44 AM
installing the latest tarball (9.6m) appears to have broken my bind

named: relocation error: /usr/lib/libdns.so.5: undefined symbol: ENGINE_new

redhat 8, any ideas?


Justin

ctnchris
10-21-2004, 10:21 AM
I tried this guide but couldnt get it to work. Is there anywhere I can grab an rpm for a newer version of openssl? Fedora legacy has 0.9.6b :\

vandal
10-21-2004, 12:09 PM
fedora legacy's is back patched and secured though.

ctnchris
10-23-2004, 10:54 AM
Is there any program I can run from the outside that will test (not ruin) my server for vulnerbilities? (i think thats spelled wrong).

markus
11-26-2004, 07:19 PM
Hi all!

I'm running a fresh version of DirectAdmin v1.23.1 (installed by DA sales) under RHE 3.

I have upgraded openssl using the following commands:

cd /usr/src
wget http://www.openssl.org/source/openssl-0.9.7e.tar.gz
tar -xvzf openssl-0.9.7e.tar.gz
cd openssl-0.9.7e/

./config --prefix=/usr no-threads shared
make
make test
make install
echo "/usr/local/ssl/lib" >> /etc/ld.so.conf
ldconfig -v

rm -f /lib/libssl.so.4
ln -s /usr/local/ssl/lib/libssl.so.0.9.7 /lib/libssl.so.4
rm -f /usr/lib/libcrypto.so
ln -s /usr/local/ssl/lib/libcrypto.so.0.9.7 /usr/lib/libcrypto.so.0

Then I have rebuild Apache+mod_ssl using the following commads:
cd /usr/local/directadmin/customapache
rm -f configure.*
./build clean
./build update
./build all

Everything seems to run ok, except the fact that my phpinfo shows different versions of openssl. :eek:

1) Apache version: Apache/1.3.33 (Unix) mod_ssl/2.8.22 OpenSSL/0.9.7e PHP/4.3.9 mod_perl/1.29 FrontPage/5.0.2.2510

2) CURL information: libcurl/7.12.2 OpenSSL/0.9.7e zlib/1.2.1

3) OpenSSL Version: OpenSSL 0.9.7a Feb 19 2003


How can I make apache use the new installed version of openssl? Did I forgot to do anything else?

Any hint would be much appreciated. Thanks