Richard Bejtlich (taosecurity at gmail dot com)
www.taosecurity.com/keeping_freebsd_applications_up-to-date.html
www.taosecurity.com/keeping_freebsd_applications_up-to-date.pdf
www.taosecurity.com/keeping_freebsd_applications_up-to-date.ps
21 April 2005
This article is a sequel to my previous work "Keeping FreeBSD Up-To-Date," available here:
www.taosecurity.com/keeping_freebsd_up-to-date.html
An important system administration task, and a principle of running a defensible network, is keeping operating systems and applications up-to-date. Running current software is critical when older services are vulnerable to exploitation. Obtaining new features not found in older applications is another reason to run current software. Fortunately, open source software offers a variety of means to give users a secure, capable computing environment.
This article presents multiple ways to keep FreeBSD applications up-to-date. I explain how to install and upgrade several applications on a FreeBSD 5.2.1 RELEASE system. In my previous article "Keeping FreeBSD Up-To-Date," I described how to patch and upgrade the FreeBSD operating system, beginning with FreeBSD 5.2.1 and ending with FreeBSD 5-STABLE. Taken as a pair, these two articles will help system administrators keep their FreeBSD OS and applications current and defensible.
I chose FreeBSD 5.2.1, released in February 2004, as my reference platform because the applications bundled with it have been updated several times in the past ten months. These updates provide good material for the case-based approach used in this article. All of the techniques explained here work on the most recent FreeBSD version, 5.3.
All of the work done in this article was done remotely via OpenSSH. One danger of performing remote upgrades is losing connection during a critical phase of the process. One software-based way to deal with this issue is to conduct all remote upgrades within a screen(1) session. [1] Should you lose connectivity during the upgrade while running screen, your session will continue uninterrupted. The screen(1) program has suffered security problems in the past, so balance its features against the possible risks.
My advice on upgrading these applications is based on deploying FreeBSD on servers, workstations, and laptops since 2000. The article represents a mix of my interpretations of official FreeBSD documentation, inputs from mentors, and the result of my own experimentation and deployment strategies. This guide cannot be anywhere near a complete reference on keeping FreeBSD applications up-to-date or maintaining secure software. I strongly recommend reading the excellent FreeBSD Handbook as well as the multiple helpful published books on FreeBSD.
Third party applications may be installed using source code, the FreeBSD ports tree, or precompiled packages. Each will be described in detail, but not exhaustively. The tips here are enough to get the novice to intermediate system administrator managing applications on FreeBSD.
If you are reading this as a hard copy, the right hand side of the page may be cut off. This is important when long URLs or command line statements are involved. Please refer to the HTML version of this document for the authoritative version.
Source code is typically packaged as an archive processed by the tar(1) and gzip(1) programs, with the .tar.gz or .tgz suffixes. In the following example, we install the Snort output reader Barnyard from source code. Until further notified, we take these actions as a user and not as root.
As a good system administration practice, one might want to create a specific directory to hold source code for third party applications. Creating /usr/local/src is the method used here.
freebsd521$ mkdir /usr/local/srcNext we download the Barnyard tarball with fetch(1), after visiting the Barnyard project page and locating a suitable distribution site. [2]
freebsd521$ fetch http://kent.dl.sourceforge.net/sourceforge/barnyard/barnyard-0.2.0.tar.gz Receiving barnyard-0.2.0.tar.gz (161543 bytes): 100% 161543 bytes transferred in 6.6 seconds (23.78 kBps)Now extract the contents of the archive:
freebsd521$ tar -xzvf barnyard-0.2.0.tar.gz barnyard-0.2.0/ barnyard-0.2.0/docs/ barnyard-0.2.0/docs/BUGS ...edited... barnyard-0.2.0/src/input-plugins/dp_stream_stat.h barnyard-0.2.0/src/input-plugins/dp_stream_stat.cOnce the source code is extracted, we change into the new directory and look for a configure script:
freebsd521$ cd barnyard-0.2.0 freebsd521$ ls AUTHORS README config.guess docs src COPYING acconfig.h config.h.in etc stamp-h.in LICENSE.QPL aclocal.m4 config.sub install-sh Makefile.am autoclean.sh configure missing Makefile.in autojunk.sh configure.in mkinstalldirsWe execute the configure script and pass an optional parameter to enable MySQL output.
freebsd521$ ./configure --enable-mysql
creating cache ./config.cache
checking for a BSD compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking whether make sets ${MAKE}... yes
checking for working aclocal-1.4... missing
...edited...
checking for strerror... yes
checking for /mysql.h... no
**********************************************
ERROR: unable to find mysql headers (mysql.h)
checked in the following places
/mysql.h
**********************************************
The configure script abruptly stops because it cannot find mysql.h. This demonstrates the major weakness of installing software from source code. Administrators must be aware of any dependencies required by the application, and address them prior to installing the new software.
Assume we take care of the dependency (using a method to be demonstrated shortly). We re-run the configure script:
freebsd521$ ./configure --enable-mysql
creating cache ./config.cache
checking for a BSD compatible install... /usr/bin/install -c
...edited...
checking for strerror... yes
checking for /usr/local/include/mysql/mysql.h... yes
checking for mysql_real_connect in -lmysqlclient... no
**********************************************
ERROR: unable to find mysqlclient library
checked in the following places
/usr/local/lib/mysql
**********************************************
Now we see a new error. Although the mysql.h file was found, the configure script is not finding the MySQL client library where it expects to find it. A look in /usr/local/lib/mysql shows the following:
freebsd521$ ls /usr/local/lib/mysql/ libmysqlclient.a libmysqlclient.so libmysqlclient.so.10It appears the files needed are in place. This error demonstrates the second weakness of installing software from source. Sometimes it does not work as expected, and administrators must tweak installation files to accommodate their systems.
The resolution to this problem surfaces by doing a Google search. If we modify the configure script as shown, it fixes the problem:
original configure script:
LIBS="${LIBS} -lz -lssl -lmysqlclient"
modified configure script:
LIBS="${LIBS} -lz -lssl -lcrypto -lmysqlclient"
Rerun the configure script:
freebsd521$ ./configure --enable-mysql creating cache ./config.cache checking for a BSD compatible install... /usr/bin/install -c ...edited... checking for strerror... (cached) yes checking for /usr/local/include/mysql/mysql.h... yes checking for mysql_real_connect in -lmysqlclient... yes ...edited... creating config.h config.h is unchangedBecause the configure script completed, we can now execute make(1) and begin compiling Barnyard:
freebsd521$ make make all-recursive Making all in src Making all in output-plugins gcc -DHAVE_CONFIG_H -I. -I. -I../.. -I../.. -I../../src -I/usr/local/include/mysql -DENABLE_MYSQL -g -O2 -Wall -c op_decode.c gcc -DHAVE_CONFIG_H -I. -I. -I../.. -I../.. -I../../src -I/usr/local/include/mysql -DENABLE_MYSQL -g -O2 -Wall -c op_fast.c ...edited... gcc -DHAVE_CONFIG_H -I. -I. -I.. -I.. -I/usr/local/include/mysql -DENABLE_MYSQL -g -O2 -Wall -c barnyard.c gcc -DHAVE_CONFIG_H -I. -I. -I.. -I.. -I/usr/local/include/mysql -DENABLE_MYSQL -g -O2 -Wall -c mstring.c ...edited... ProgVars.c:672: warning: long unsigned int format, time_t arg (arg 3) gcc -g -O2 -Wall -L/usr/local/lib/mysql -o barnyard barnyard.o mstring.o strlcatu.o strlcpyu.o util.o spool.o sid.o debug.o classification.o CommandLineArgs.o ConfigFile.o ProgVars.o output-plugins/libop.a input-plugins/libdp.a -lz -lssl -lcrypto -lmysqlclientAt this point we assume root privileges and execute 'make install' to copy the Barnyard executable to /usr/local/bin:
freebsd521# make install Making install in src Making install in output-plugins Making install in input-plugins /bin/sh ../mkinstalldirs /usr/local/bin /usr/bin/install -c barnyard /usr/local/bin/barnyardOn FreeBSD systems where the root shell is tcsh or csh, administrators must run rehash(1) to recompute the hash table for the PATH variable. If one does not do this, the system will not find barnyard(1). For example:
freebsd521# barnyard -h barnyard: Command not found. freebsd521# ls /usr/local/bin/barnyard /usr/local/bin/barnyard freebsd521# rehash freebsd521# barnyard -h Barnyard Version 0.2.0 (Build 32) Usage: barnyard [OPTIONS]... (continual mode) or: barnyard -o [OPTIONS]... FILES... (batch mode) ...truncated...Barnyard is now installed.
Installation from source code has four main weaknesses:
FreeBSD offers an extremely powerful means of installing software. This system is known as the ports tree. Administrators have the option of adding this structure to their system during installation. Alternatively, a recent copy of the source tree archive can be downloaded from www.freebsd.org/ports/ and extracted to the /usr/ports directory.
The FreeBSD ports tree is a system to facilitate the installation of third party applications. The FreeBSD Handbook and other references already comprehensively explain this resource, so I will restrict this text to demonstrating the installation of a tool, FreeBSD Update, using the ports tree. In the previous article we used FreeBSD Update to keep the OS up-to-date. As such it may be considered a "security tool."
A quick look at /usr/ports shows multiple files and directories, corresponding to categories of tools:
freebsd521# cd /usr/ports freebsd521# ls .cvsignore audio finance math shells INDEX benchmarks french mbone sysutils INDEX-5 biology ftp misc textproc LEGAL cad games multimedia ukrainian MOVED chinese german net vietnamese Makefile comms graphics news www Mk converters hebrew palm x11 README databases hungarian picobsd x11-clocks README.html deskutils irc polish x11-fm Templates devel japanese portuguese x11-fonts Tools distfiles java print x11-servers arabic dns korean russian x11-toolkits archivers editors lang science x11-wm astro emulators mail securityA quick look inside the security directory shows tools which provide security functions:
freebsd521# ls security | head ADMsmb ADMsnmp IMHear Makefile README.html aafid2 acid aescrypt aide altivoreIn fact, we see 'freebsd-update' listed, if we use grep(1) to narrow down the directory listing:
freebsd521# ls security | grep -i update freebsd-updateIf we were not able to guess at the location of our tool of interest, we could turn to other resources. Two Web-based options exist. Search functions at www.freshports.org or www.freebsd.org/ports reveal that the FreeBSD Update program is found in the security/freebsd-update portion of the tree.
The ports tree itself offers two other methods to find tools of interest. From the /usr/ports directory, use make(1) to find ports with the "update" keyword:
freebsd521# make search key=update | more ...edited... Port: freebsd-update-1.4 Path: /usr/ports/security/freebsd-update Info: Fetches and installs binary updates to FreeBSD Maint: cperciva@daemonology.net Index: security B-deps: R-deps: bsdiff-4.1 ...truncated...If we were sure of the tool's name (i.e., "freebsd-update"), we could leverage that knowledge directly to check if the tool is in the ports tree:
freebsd521# make search name=freebsd-update Port: freebsd-update-1.4 Path: /usr/ports/security/freebsd-update Info: Fetches and installs binary updates to FreeBSD Maint: cperciva@daemonology.net Index: security B-deps: R-deps: bsdiff-4.1At this point we've used several means to locate freebsd-update(1). Now we install it as user root by running make(1) inside the /usr/ports/security/freebsd-update directory. Assume you are taking this action just after FreeBSD 5.2.1 was released, so the freebsd-update(1) version shown (1.4) is the newest available:
freebsd521# cd security/freebsd-update/ freebsd521# make >> freebsd-update-1.4.tar.gz doesn't seem to exist in /usr/ports/distfiles/. >> Attempting to fetch from http://www.daemonology.net/freebsd-update/. Receiving freebsd-update-1.4.tar.gz (29567 bytes): 100% 29567 bytes transferred in 0.1 seconds (236.29 kBps) ===> Extracting for freebsd-update-1.4 >> Checksum OK for freebsd-update-1.4.tar.gz. /usr/bin/sed -e "s#PREFIX=/usr/local#PREFIX=/usr/local#g" /usr/ports/security/freebsd-update/work/freebsd-update-1.4/freebsd-update > /usr/ports/security/freebsd-update/work/freebsd-update-1.4/freebsd-update.new /bin/mv /usr/ports/security/freebsd-update/work/freebsd-update-1.4/freebsd-update.new /usr/ports/security/freebsd-update/work/freebsd-update-1.4/freebsd-update ===> Patching for freebsd-update-1.4 ===> Configuring for freebsd-update-1.4 ===> Building for freebsd-update-1.4 cc -O -pipe -mcpu=pentiumpro -mcpu=pentiumpro -O3 -I lib -o freebsd-update-verify verify.c lib/hashtab.c lib/hash.c lib/fftlut.c lib/fft.c lib/smpa.c lib/numt.c lib/rsa.cWe see the freebsd-update-1.4.tar.gz source code archive is retrieved from www.daemonology.net/freebsd-update. The archive is extracted, patched, configured, and compiled. Next we run 'make install' to install the program:
freebsd521# make install ===> Installing for freebsd-update-1.4 ===> freebsd-update-1.4 depends on executable: bspatch - not found ===> Verifying install for bspatch in /usr/ports/misc/bsdiff >> bsdiff-4.1.tar.gz doesn't seem to exist in /usr/ports/distfiles/. >> Attempting to fetch from http://www.daemonology.net/bsdiff/. Receiving bsdiff-4.1.tar.gz (7721 bytes): 100% 7721 bytes transferred in 0.0 seconds (156.32 kBps) ===> Extracting for bsdiff-4.1 >> Checksum OK for bsdiff-4.1.tar.gz. ===> Patching for bsdiff-4.1 ===> Configuring for bsdiff-4.1 ===> Building for bsdiff-4.1 cc -O -pipe -mcpu=pentiumpro -mcpu=pentiumpro -O3 bsdiff.c -o bsdiff cc -O -pipe -mcpu=pentiumpro -mcpu=pentiumpro -O3 bspatch.c -o bspatch ===> Installing for bsdiff-4.1 ===> Generating temporary packing list ===> Checking if misc/bsdiff already installed install -c -s -m 555 bsdiff bspatch /usr/local/bin install -c -m 444 bsdiff.1 bspatch.1 /usr/local/man/man1 ===> Compressing manual pages for bsdiff-4.1 ===> Registering installation for bsdiff-4.1 ===> Returning to build of freebsd-update-1.4 ===> Generating temporary packing list ===> Checking if security/freebsd-update already installed install -s -o root -g wheel -m 555 freebsd-update-verify /usr/local/sbin install -o root -g wheel -m 555 freebsd-update /usr/local/sbin install -o root -g wheel -m 444 freebsd-update.conf.5 /usr/local/man/man5 install -o root -g wheel -m 444 freebsd-update.8 /usr/local/man/man8 mkdir /usr/local/freebsd-update install -o root -g wheel -m 444 freebsd-update.conf /usr/local/etc/freebsd-update.conf.sample mkdir /usr/local/share/doc/freebsd-update install -o root -g wheel -m 444 LICENSE README VERSION /usr/local/share/doc/freebsd-update ...edited... ===> Compressing manual pages for freebsd-update-1.4 ===> Registering installation for freebsd-update-1.4During this process, something interested happened. The installation process recognized that freebsd-update(1) had an unresolved dependency. Freebsd-update(1) requires the bsdiff(1) program, but bsdiff(1) was not installed. Thanks to the power of the ports tree, the installation process first installed bsdiff(1) and then continued with the installation of freebsd-update(1). If bsdiff(1) had any unresolved dependencies, the installation process would have taken care of those before declaring bsdiff(1) ready. This incredible ports tree feature makes installing software much simpler than compiling source code manually.
When done, the new applications are installed as packages. This is a crucial point to understand. Any application installed through the ports tree, or as a precompiled package (shown next), ends up as a package on the system. For example, the native pkg_info(1) tool reveals the packages installed on our test system:
freebsd521# pkg_info bash-2.05b.007 The GNU Bourne Again Shell bsdiff-4.1 Generates and applies patches to binary files freebsd-update-1.4 Fetches and installs binary updates to FreeBSD mysql-client-3.23.58 Multithreaded SQL database (client) perl-5.6.1_15 Practical Extraction and Report LanguageInstallation using the ports tree has three main weaknesses:
While installing Barnyard from source code, we came across a dependency for the MySQL client. This section shows how I quickly resolved this dependency by installing the MySQL client as a precompiled package.
Each FreeBSD release (at least for i386) is shipped with a set of packages. These are found on CD-ROMs shipped by vendors like FreeBSDMall.com or on the FreeBSD FTP mirrors. For example, the following FTP directory on FTP mirror nine contains packages for the i386 platform:
ftp://ftp9.freebsd.org/pub/FreeBSD/ports/i386/packages-5.2.1-releaseWithin that directory, the 'All' directory shows the actual packages:
ftp> cd All 250 "/pub/FreeBSD/ports/i386/packages-5.2-release/All" is new cwd. ftp> ls freebsd-update* 227 Entering Passive Mode (128,10,252,13,223,67) 150 Data connection accepted from 69.243.15.208:49163; transfer starting. -rw-r--r-- 1 ftpuser ftpusers 28927 Dec 9 2003 freebsd-update-1.4.tbz 226 Listing completed.The 'Latest' directory is a collection of symbolic links using the base name for each package. Here we see that Latest/freebsd-update.tbz is a link to the real package, which bears a version number of 1.4. This is the same version we installed using the ports tree.
ftp> cd .. 250 "/pub/FreeBSD/ports/i386/packages-5.2-release" is new cwd. ftp> cd Latest 250 "/pub/FreeBSD/ports/i386/packages-5.2-release/Latest" is new cwd. ftp> ls freebsd-update* 227 Entering Passive Mode (128,10,252,13,223,69) 150 Data connection accepted from 69.243.15.208:49164; transfer starting. lrwxrwxrwx 1 ftpuser ftpusers 29 Aug 26 20:58 freebsd-update.tbz -> ../All/freebsd-update-1.4.tbz 226 Listing completed.Returning to our problem of resolving Barnyard's dependency on the MySQL client, we can install the program using the pkg_add(1) command. The -v switch enables verbose output and the -r switch denotes fetching the package from a remote location. If we had already downloaded the package locally, -r is not needed. I prefer to install all packages using -r, because the installation process will also download any dependencies:
freebsd521# pkg_add -vr mysql-client looking up ftp.freebsd.org connecting to ftp.freebsd.org:21 setting passive mode opening data connection initiating transfer Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-5.2.1-release/Latest/mysql-client.tbz...+CONTENTS +COMMENT +DESC +MTREE_DIRS man/man1/isamchk.1.gz man/man1/isamlog.1.gz man/man1/mysql.1.gz ...edited... lib/mysql/libmysqlclient.so.10 tar command returns 0 status Done. Package 'mysql-client-3.23.58' depends on 'perl-5.6.1_15' with 'lang/perl5' origin. - already installed. extract: Package name is mysql-client-3.23.58 extract: CWD to /usr/local extract: /usr/local/man/man1/isamchk.1.gz extract: /usr/local/man/man1/isamlog.1.gz ...edited... extract: /usr/local/lib/mysql/libmysqlclient.so.10 extract: execute '/sbin/ldconfig -m /usr/local/lib/mysql' extract: CWD to . Running mtree for mysql-client-3.23.58.. mtree -U -f +MTREE_DIRS -d -e -p /usr/local >/dev/null Attempting to record package into /var/db/pkg/mysql-client-3.23.58.. Trying to record dependency on package 'perl-5.6.1_15' with 'lang/perl5' origin. Package mysql-client-3.23.58 registered in /var/db/pkg/mysql-client-3.23.58First the pkg_add(1) process retrieves the mysql-client.tbz package from the "All" directory. This package name is really a symlink to the mysql-client-3.23.58.tbz package.
The package installation process also resolves dependencies automatically. The mysql-client-3.23.58 package depends on perl-5.6.1_15. Since Perl is already installed, the package deployment continues. When done, we have our MySQL client and Barnyard can be installed from source code.
Installation using precompiled packages has five main weaknesses:
The most direct way to update an application installed from source code is to uninstall the old version and install the new version. First, execute 'make uninstall' in the directory from which you executed 'make install'. For example:
freebsd521# cd /usr/local/src/barnyard-0.2.0 freebsd521# make uninstall Making uninstall in src Making uninstall in output-plugins Making uninstall in input-plugins list='barnyard'; for p in $list; do rm -f /usr/local/bin/`echo $p|sed 's/$//'|sed 's,x,x,'|sed 's/$//'`; done freebsd521# ls /usr/local/bin/barnyard ls: /usr/local/bin/barnyard: No such file or directoryNow, download the new version of Barnyard, extract it, and follow the installation instructions already posted.
Now we turn to keeping packages up-to-date. There is no section on keeping "ports" up-to-date or "packages" up-to-date. To reiterate, applications installed using the ports tree or precompiled packages all end up as packages on the system. While we will see tools with "port" or "pkg" in their names, all tend to act on packages installed on FreeBSD.
The most direct way to "update" a package is to remove it and install a new version. Early we used pkg_info to show installed packages. A check in /var/db/pkg also shows the packages installed.
freebsd521# cd /var/db/pkg freebsd521# ls bash-2.05b.007 freebsd-update-1.4 perl-5.6.1_15 bsdiff-4.1 mysql-client-3.23.58I prefer to use pkg_delete(1) from this directory, because I can use tab-completion to specify the entire package name:
freebsd521# pkg_delete -v mysql-client-3.23.58 Change working directory to /usr/local Delete file /usr/local/man/man1/isamchk.1.gz Delete file /usr/local/man/man1/isamlog.1.gz ...edited... Delete file /usr/local/man/man1/safe_mysqld.1.gz Execute 'rm -f /usr/local/man/cat1/isamchk.1 /usr/local/man/cat1/isamchk.1.gz' Execute 'rm -f /usr/local/man/cat1/isamlog.1 /usr/local/man/cat1/isamlog.1.gz' ...edited... Execute 'rm -f /usr/local/man/cat1/safe_mysqld.1 /usr/local/man/cat1/safe_mysqld.1.gz' Delete file /usr/local/bin/mysql Delete file /usr/local/bin/mysqladmin ...edited... Delete file /usr/local/lib/mysql/libmysqlclient.so.10 Delete directory /usr/local/include/mysql Delete directory /usr/local/lib/mysql Execute 'if [ -f /usr/local/info/dir ]; then if sed -e '1,/Menu:/d' /usr/local/info/dir | grep -q '^[*] '; then true; else rm /usr/local/info/dir; fi; fi' Execute '/sbin/ldconfig -R' Change working directory to . Trying to remove dependency on package 'perl-5.6.1_15' with 'lang/perl5' origin.The package is gone, according to pkg_info(1) and a listing of /var/db/pkg:
freebsd521# pkg_info bash-2.05b.007 The GNU Bourne Again Shell bsdiff-4.1 Generates and applies patches to binary files freebsd-update-1.4 Fetches and installs binary updates to FreeBSD perl-5.6.1_15 Practical Extraction and Report Language freebsd521# ls bash-2.05b.007 freebsd-update-1.4 bsdiff-4.1 perl-5.6.1_15Now that the package is gone, we must look for a newer version. The package we deleted, mysql-client-3.23.58, was the version shipped with FreeBSD 5.2.1 RELEASE. It was found in the following FTP directory. We show the contents of FTP mirror 9, but you are free to use whatever mirror you like;
ftp://ftp9.freebsd.org/pub/FreeBSD/ports/i386/packages-5.2.1-release/AllPackages compiled from the latest source code on the most recent version of FreeBSD, 5-STABLE, are found here:
ftp://ftp9.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/LatestIn this directory we find a new version of the MySQL client, namely mysql-client-3.23.58_3.tbz. We can install this version using pkg_add(1) if we change our PACKAGESITE environment variable:
freebsd521# setenv PACKAGESITE ftp://ftp9.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/Latest/We check to see what packages are available there:
ftp> pwd 257 "/pub/FreeBSD/ports/i386/packages-5-stable/Latest" is cwd. ftp> ls mysql*client* 227 Entering Passive Mode (128,10,252,13,224,196) 150 Data connection accepted from 69.243.15.208:49184; transfer starting. lrwxrwxrwx 1 ftpuser ftpusers 33 Nov 17 21:01 mysql323-client.tbz -> ../All/mysql-client-3.23.58_3.tbz lrwxrwxrwx 1 ftpuser ftpusers 30 Nov 17 21:01 mysql40-client.tbz -> ../All/mysql-client-4.0.22.tbz lrwxrwxrwx 1 ftpuser ftpusers 29 Nov 17 21:01 mysql41-client.tbz -> ../All/mysql-client-4.1.7.tbz lrwxrwxrwx 1 ftpuser ftpusers 29 Nov 17 21:01 mysql50-client.tbz -> ../All/mysql-client-5.0.1.tbz 226 Listing completed.Notice that the various MySQL clients now all have version numbers. We are interested in using mysql323-client.tbz with Barnyard.
Now we use pkg_add(1) to install the newest MySQL 3.x client listed in the STABLE package directory:
freebsd521# pkg_add -vr mysql323-client looking up ftp9.freebsd.org connecting to ftp9.freebsd.org:21 setting passive mode opening data connection initiating transfer Fetching ftp://ftp9.freebsd.org/pub/FreeBSD/ports/i386/packages-5-stable/Latest/mysql323-client.tbz... ...edited... extract: /usr/local/lib/mysql/libmysqlclient_r.so.10 extract: execute '/sbin/ldconfig -m /usr/local/lib/mysql' extract: CWD to . Running mtree for mysql-client-3.23.58_3.. mtree -U -f +MTREE_DIRS -d -e -p /usr/local >/dev/null Attempting to record package into /var/db/pkg/mysql-client-3.23.58_3.. Package mysql-client-3.23.58_3 registered in /var/db/pkg/mysql-client-3.23.58_3When done, the new version is installed:
freebsd521# pkg_info | grep mysql mysql-client-3.23.58_3 Multithreaded SQL database (client)This process seems fairly simple, but there are problems. First, we had to manually verify that a new version of the MySQL client was available. Then we deleted it and reinstalled it. If any other packages (not source code like Barnyard) required mysql-client as a dependency, pkg_delete(1) would have complained and not let us delete mysql-client. We could have forced deinstallation, but that's a sloppy system administration practice.
Should other applications have required the MySQL client, we could have deleted them, then deleted mysql-client, and reinstalled everything. Again, that is a lot of work. Fortunately, alternatives (described next) exist.
Incidentally, you can use sysutils/pkg_tree to see package dependencies. Here we see what dependencies freebsd-update has:
freebsd521# pkg_tree freebsd-update freebsd-update-1.4 \__ bsdiff-4.1We now know that if we tried to pkg_delete(1) the bsdiff package, the system would complain because freebsd-update(1) depends on bsdiff(1):
freebsd521# cd /var/db/pkg freebsd521# pkg_delete bsdiff-4.1/ pkg_delete: package 'bsdiff-4.1' is required by these other packages and may not be deinstalled: freebsd-update-1.4A tool one can use to trim installed packages is sysutils/pkg_cutleaves. This tool begins with "leaf" packages, asking if you want to remove them, followed by the packages upon which they depend. In the following example, we hit [return] whenever we see a package we want to keep. When we come to nmap(1), we decide to remove it with 'd':
drury:/root# pkg_cutleaves Package 1 of 42: XFree86-4.3.0,1 - X11/XFree86 core distribution (complete, using mini/meta-ports) XFree86-4.3.0,1 - [keep]/(d)elete/(f)lush marked pkgs/(a)bort? ** Keeping XFree86-4.3.0,1. ...edited... Package 21 of 42: nmap-3.77 - Port scanning utility for large networks nmap-3.77 - [keep]/(d)elete/(f)lush marked pkgs/(a)bort? d ** Marking nmap-3.77 for removal. ...edited... Deleting nmap-3.77 (package 1 of 1). ---> Deinstalling 'nmap-3.77' [Updating the pkgdb <format:bdb1_btree> in /var/db/pkg ... - 104 packages found (-1 +0) (...) done]At this point, pkg_cutleaves(1) asks if we want to continue. If we do, we have the option of deleting a nmap(1) dependency, pcre:
Go on with new leaf packages ((y)es/[no])? y Package 1 of 1: pcre-5.0 - Perl Compatible Regular Expressions library pcre-5.0 - [keep]/(d)elete/(f)lush marked pkgs/(a)bort? d ** Marking pcre-5.0 for removal. Deleting pcre-5.0 (package 1 of 1). ---> Deinstalling 'pcre-5.0' [Updating the pkgdb <format:bdb1_btree> in /var/db/pkg ... - 103 packages found (-1 +0) (...) done] ** Didn't find any new leaves to work with, exiting. ** Deinstalled packages: nmap-3.77 pcre-5.0 ** Number of deinstalled packages: 2Since there are no longer any dependencies for nmap(1), the process ends. Using a tool like pkg_cutleaves(1) allows us to trim down unnecessary packages very easily.
The first step towards better package management is updating the FreeBSD ports tree. The contents of /usr/ports will remain the same as they are on the installation CD unless an administrator updates them. To let the system know what port versions are available, we must update the ports tree. This differs from other systems that contact a third party server to check for updates. Without updating the ports tree, the system only knows if installed packages are what it thinks of as "up-to-date."
We start by using pkg_version(1) to show what the system thinks of its package status:
freebsd521# pkg_version -v bash-2.05b.007 = up-to-date with port bsdiff-4.1 = up-to-date with port cvsup-without-gui-16.1h = up-to-date with port freebsd-update-1.4 = up-to-date with port mysql-client-3.23.58_3 > succeeds port (port has 3.23.58) perl-5.6.1_15 = up-to-date with portThe odd results for mysql-client.3.23.58_3 are a result of deinstalling the package shipped with FreeBSD 5.2.1 and installing a new package for FreeBSD 5-STABLE. The pkg_version(1) tool sees the MySQL client version is newer than that found in the ports tree. It does this by checking version information in the Makefile found in the ports tree. If that information is unavailable, it looks in the /usr/ports/INDEX-5 file (explained below).
The tool used to keep the ports tree up-to-date is the same tool used to keep the FreeBSD source tree up-to-date -- CVSup.
Here we add the GUI-less version of CVSup using pkg_add(1). Note we are adding the package that ships with FreeBSD 5.2.1 RELEASE, not the version available in the 5-STABLE tree.
freebsd521# pkg_add -vr cvsup-without-gui looking up ftp.freebsd.org connecting to ftp.freebsd.org:21 setting passive mode opening data connection initiating transfer Fetching ftp://ftp.freebsd.org/pub/FreeBSD/ports/i386/packages-5.2.1-release/Latest/cvsup-without-gui.tbz... ...edited... extract: /usr/local/share/cvsup/License extract: CWD to . Running mtree for cvsup-without-gui-16.1h.. mtree -U -f +MTREE_DIRS -d -e -p /usr/local >/dev/null Attempting to record package into /var/db/pkg/cvsup-without-gui-16.1h.. Package cvsup-without-gui-16.1h registered in /var/db/pkg/cvsup-without-gui-16.1hWe must provide a suitable configuration file for CVSup, so we create /usr/local/etc/ports-supfile with the following contents:
*default host=cvsup9.FreeBSD.org *default base=/var/db *default prefix=/usr *default release=cvs tag=. *default delete use-rel-suffix *default compress ports-allNotice the CVS tag is "." (dot). This shows that the ports tree is not tied to any particular FreeBSD version. While we could replace the . with a specific date, it makes more sense to upgrade to the latest ports tree available.
Before running CVSup, we should take a look at the two INDEX files in /usr/ports. The INDEX file is for FreeBSD 4.x, while the INDEX-5 is for FreeBSD 5.x. A look at the entry for nmap(1) in each is illustrative of the file's purpose:
freebsd521# ls -al IND* -rw-r--r-- 1 root wheel 4251062 Nov 15 2003 INDEX -rw-r--r-- 1 root wheel 4436222 Feb 14 2004 INDEX-5 freebsd521# grep "^nmap-" INDEX nmap-3.48_1|/usr/ports/security/nmap|/usr/local|Port scanning utility for large networks|/usr/ports/security/nmap/pkg-descr|eik@FreeBSD.org|security ipv6| pcre-4.4|pcre-4.4|http://www.insecure.org/nmap/ freebsd521# grep "^nmap-" INDEX-5 nmap-3.48_1|/usr/ports/security/nmap|/usr/local|Port scanning utility for large networks|/usr/ports/security/nmap/pkg-descr|eik@FreeBSD.org|security ipv6| pcre-4.4|pcre-4.4|http://www.insecure.org/nmap/Each entry shows the port name, followed by its location in the ports tree, a description, port maintainer, category, dependencies, and author URL. If there were different information for each version of FreeBSD, we would see it. Here there are no differences.
Now we run CVSup to sync our local copy of the ports tree with the version on the CVSup9 mirror. Again, this is an arbitrary choice of mirrors. The process makes every file in our /usr/ports directory the same as that on the CVSup9 mirror:
freebsd521# cvsup -g -L 2 /usr/local/etc/ports-supfile Parsing supfile "/usr/local/etc/ports-supfile" Connecting to cvsup9.FreeBSD.org Connected to cvsup9.FreeBSD.org Server software version: SNAP_16_1h Negotiating file attribute support Exchanging collection information Establishing multiplexed-mode data connection Running Updating collection ports-all/cvs Checkout ports/CHANGES Delete ports/INDEX Delete ports/INDEX-5 ...edited... Edit ports/Tools/scripts/plist Add delta 1.6 2004.02.27.21.01.02 green Edit ports/Tools/scripts/release/doit.sh Add delta 1.5 2004.06.08.21.57.01 murray Edit ports/Tools/scripts/release/scrubindex.pl Add delta 1.2 2004.06.08.21.57.01 murray Checkout ports/Tools/scripts/security-check.awk Checkout ports/Tools/scripts/sunshar/Makefile ...edited... Delete ports/x11-wm/xwmm/pkg-plist Edit ports/x11-wm/yawm/distinfo Add delta 1.2 2004.01.27.16.12.36 trevor Applying fixups for collection ports-all/cvs Fixup ports/textproc/xerces-c2/files/patch-ab Shutting down connection to server Finished successfullyOn a FreeBSD 5.2.1 installation, this process takes a while. CVSup has to merge any changes between the ports tree of a 5.2.1 system with the latest version available from FreeBSD.org.
Notice that the process deleted the INDEX and INDEX-5 files. The easiest way to rebuild the INDEX-5 file is to execute 'make fetchindex' in /usr/ports:
freebsd521# make fetchindex Receiving INDEX-5.bz2 (612684 bytes): 100% 612684 bytes transferred in 2.0 seconds (292.50 kBps) freebsd521# ls -al INDEX-5 -rw-r--r-- 1 root wheel 5974063 Dec 23 15:09 INDEX-5A FreeBSD 5.x system does not need the INDEX file used by FreeBSD 4.x.
An alternative to using 'make fetchindex' is Matthew Seaman's tool Portindex (sysutils/p5-FreeBSD-Portindex). I find 'make fetchindex' suits my needs, so I do not use Portindex.
An imporant file to reference whenever you consider upgrading installed applications is /usr/ports/UPDATING.
For example, here is the beginning of that file at the time of writing this article:
This file documents some of the problems you may encounter when upgrading your ports. We try our best to minimize these disruptions, but sometimes they are unavoidable. You should get into the habit of checking this file for changes each time you update your ports collection, before attempting any port upgrades. 20041222: AFFECTS: users of security/clamav, security/clamav-devel AUTHOR: jylefort@brutele.be The ClamAV database path has changed from /usr/local/share/clamav to /var/db/clamav. You should update the DatabaseDirectory keyword in /usr/local/etc/clamd.conf and /usr/local/etc/freshclam.conf.If you carefully heed the advice in UPDATING your upgrades will proceed more smoothly.
Now that our ports tree is up-to-date, we can use pkg_version(1) to see what packages require updating:
freebsd521# pkg_version -v bash-2.05b.007 < needs updating (port has 2.05b.007_2) bsdiff-4.1 < needs updating (port has 4.2) cvsup-without-gui-16.1h = up-to-date with port freebsd-update-1.4 < needs updating (port has 1.6_1) mysql-client-3.23.58_3 = up-to-date with port perl-5.6.1_15 = up-to-date with portIt seems several are outdated. Earlier we showed how to use pkg_delete(1) to remove an outdate MySQL client, and pkg_add(1) to install a precompiled package of a newer version. Here we'll demonstrate a method using the ports tree to update bash(1).
First we deinstall shells/bash2:
freebsd521# cd /usr/ports/shells/bash2 freebsd521# make deinstall ===> Deinstalling for shells/bash2 ===> Deinstalling bash-2.05b.007 updating /etc/shellsNow reinstall it:
freebsd521# make reinstall ===> Vulnerability check disabled => bash-2.05b.tar.gz doesn't seem to exist in /usr/ports/distfiles/. => Attempting to fetch from http://ftp.gnu.org/gnu/bash/. Receiving bash-2.05b.tar.gz (1956216 bytes): 78% ...edited... ===> Compressing manual pages for bash-2.05b.007_2 ===> Registering installation for bash-2.05b.007_2 freebsd521# pkg_version -v | grep bash bash-2.05b.007_2 = up-to-date with portFor this example I chose a port that did not have any dependencies to satisfy. Like the earlier example showing the pkg_delete(1) and pkg_add(1) of the MySQL client, this approach isn't as simple as one might like.
The best method created thus far to keep packages up-to-date requires the Portupgrade tool. You can install it using the ports tree as it is found in sysutils/portupgrade, or you can install it as a precompiled package. On a slow system, I recommend installing Portupgrade with the precompiled package. Portupgrade's Ruby dependencies are fairly heavy.
Portupgrade is well-documented, so here I chose to show how I use it. [3] This allows you to compare Portupgrade to the methods already demonstrated.
When you install Portupgrade, you will receive several tools that perform functions similar to those found in the native package management tools. The first is portsdb(1), which builds the INDEX.db file used by Portupgrade:
janney:/usr/ports# portsdb -u [Updating the portsdb <format:bdb1_btree> in /usr/ports ... - 12144 port entries found ...............2000.........3000.........4000.........5000.........6000.........7000... ......8000.........9000.........10000.........11000.........12000. ..... done]Note: When writing this article using FreeBSD 5.2.1, the version of portsdb(1) packaged with Portupgrade did not like the format of the INDEX-5 created by 'make fetchindex'. I recommend using an updated version of the Portupgrade suite to keep your applications up-to-date. If you are using FreeBSD 5.3 RELEASE or higher, you should not have a problem. I have not encountered any problems with 'make fetchindex' on FreeBSD 5.3.
Here is the new file:
janney:/usr/ports# ls -al INDEX.db -rw-r--r-- 1 root wheel 12296192 Dec 23 18:23 INDEX.dbWe now use the portversion(1) tool to see which packages need updating, similar to using pkg_version(1):
freebsd521# portversion -v [Rebuilding the pkgdb <format:bdb1_btree> in /var/db/pkg ... - 10 packages found (-0 +10) .......... done] bash-2.05b.007_2 = up-to-date with port bsdiff-4.1 < needs updating (port has 4.2) cvsup-without-gui-16.1h = up-to-date with port freebsd-update-1.4 < needs updating (port has 1.6_1) mysql-client-3.23.58_3 = up-to-date with port perl-5.6.1_15 = up-to-date with port portupgrade-20030723 < needs updating (port has 20041224) ruby-1.6.8.2003.10.15 < needs updating (port has 1.6.8.2004.07.28_1) ruby-bdb1-0.2.1 < needs updating (port has 0.2.2) ruby-shim-ruby18-1.8.1.p2 < needs updating (port has 1.8.1.p3)At this point, before actually updating these packages, I recommend consulting /usr/ports/UPDATING. Watch for advice on any of the packages you need to update. Here we find information on Ruby and Portupgrade:
20040226:
AFFECTS: i386 users of ruby and portupgrade
AUTHOR: knu@FreeBSD.org
Change the default version of ruby to 1.8 for i386.
If you are a ruby developer and want to keep ruby 1.6 as default,
please add RUBY_DEFAULT_VER=1.6 to /etc/make.conf.
Otherwise, please run the following series of commands to migrate to
ruby 1.8:
1) Reinstall portupgrade manually (and as a result ruby 1.8 will be
installed):
pkg_delete portupgrade-\*
(cd /usr/ports/sysutils/portupgrade; make install clean)
2) Reinstall everything that depends on ruby 1.6 to use ruby 1.8
instead:
portupgrade -fr lang/ruby16
3) Reinstall ruby 1.8 (because the previous step kills symlinks):
portupgrade -f lang/ruby18
4) Deinstall ruby 1.6 stuff (if you are paranoia):
pkg_deinstall -ri lang/ruby16
5) If the above commands do now work somehow and portupgrade starts
causing LoadError, please reinstall portupgrade manually again.
Whenever you get confused, you can always deinstall portupgrade
and all the ruby stuff (run "pkg_delete -r ruby-\*") and
reinstall portupgrade as a last resort.
This is very important advice, which we follow in the next example. Normally we do not have to observe such seemingly convoluted instructions to use Portupgrade to update our installed applications. However, when the package to be updated is Portupgrade itself, or one of its dependencies, extra attention is required.
We begin following the aforementioned instructions by removing Portupgrade from the system:
freebsd521# cd /var/db/pkg freebsd521# pkg_delete portupgrade-20030723/Now we reinstall Portupgrade and see that Ruby 1.8 is installed as a dependency for the new version:
freebsd521# cd /usr/ports/sysutils/portupgrade/ freebsd521# make install ===> Vulnerability check disabled => pkgtools-20041224.tar.bz2 doesn't seem to exist in /usr/ports/distfiles/. => Attempting to fetch from ftp://ftp.iDaemons.org/pub/distfiles/. Receiving pkgtools-20041224.tar.bz2 (104308 bytes): 100% 104308 bytes transferred in 3.7 seconds (27.36 kBps) ===> Extracting for portupgrade-20041224 => Checksum OK for pkgtools-20041224.tar.bz2. ===> portupgrade-20041224 depends on file: /usr/local/bin/ruby18 - not found ===> Verifying install for /usr/local/bin/ruby18 in /usr/ports/lang/ruby18 Dependency warning: used OpenSSL version contains known vulnerabilities Please update or define either WITH_OPENSSL_BASE or WITH_OPENSSL_PORT *** Error code 1 Stop in /usr/ports/lang/ruby18. *** Error code 1 Stop in /usr/ports/sysutils/portupgrade.Here we have encountered a problem because we are running an unpatched FreeBSD 5.2.1 RELEASE system. Thanks to our knowledge gained from the previous article, we use FreeBSD Update to take care of the reported OpenSSL vulnerability, then try installing Portupgrade again.
freebsd521# cd /usr/ports/sysutils/portupgrade/
freebsd521# make install
===> Vulnerability check disabled
===> Extracting for portupgrade-20041224
=> Checksum OK for pkgtools-20041224.tar.bz2.
===> portupgrade-20041224 depends on file: /usr/local/bin/ruby18 - not found
===> Verifying install for /usr/local/bin/ruby18 in /usr/ports/lang/ruby18
===> Vulnerability check disabled
=> ruby-1.8.2-preview4.tar.gz doesn't seem to exist in /usr/ports/distfiles/ruby.
=> Attempting to fetch from ftp://ftp.iij.ad.jp/pub/lang/ruby/1.8/.
Receiving ruby-1.8.2-preview4.tar.gz (3602003 bytes): 20%
...edited...
===> Registering installation for portupgrade-20041224
freebsd521# portupgrade -vfr lang/ruby16
---> Session started at: Thu, 23 Dec 2004 20:22:11 -0500
[Updating the pkgdb <format:bdb1_btree> in /var/db/pkg ... - 13 packages found (-1 +4) (...).... done]
---> Upgrade of lang/ruby16 started at: Thu, 23 Dec 2004 20:22:35 -0500
---> Upgrading 'ruby-1.6.8.2003.10.15' to 'ruby-1.6.8.2004.07.28_1' (lang/ruby16)
---> Build of lang/ruby16 started at: Thu, 23 Dec 2004 20:22:35 -0500
---> Building '/usr/ports/lang/ruby16'
===> Cleaning for ruby-1.6.8.2004.07.28_1
===> Vulnerability check disabled
=> ruby-1.6.8-2004.07.28.tar.bz2 doesn't seem to exist in /usr/ports/distfiles/ruby.
=> Attempting to fetch from ftp://ftp.iij.ad.jp/pub/lang/ruby/snapshots/.
...edited...
===> Installing for ruby18-bdb1-0.2.2
===> ruby18-bdb1-0.2.2 depends on file: /usr/local/bin/ruby18 - found
===> Generating temporary packing list
install -c -p -m 0755 bdb1.so /usr/local/lib/ruby/site_ruby/1.8/i386-freebsd5
===> Registering installation for ruby18-bdb1-0.2.2
===> Cleaning for ruby-1.8.2.p4
===> Cleaning for ruby18-bdb1-0.2.2
---> Removing temporary backup files
---> Installation of databases/ruby-bdb1 ended at: Thu, 23 Dec 2004 20:35:23 -0500 (consumed 00:00:15)
---> Cleaning out obsolete shared libraries
[Updating the pkgdb <format:bdb1_btree> in /var/db/pkg ... - 12 packages found (-1 +1) (...). done]
---> Upgrade of databases/ruby-bdb1 ended at: Thu, 23 Dec 2004 20:35:30 -0500 (consumed 00:01:02)
---> Listing the results (+:done / -:ignored / *:skipped / !:failed)
+ lang/ruby16 (ruby-1.6.8.2003.10.15)
+ lang/ruby16-shim-ruby18 (ruby-shim-ruby18-1.8.1.p2)
+ databases/ruby-bdb1 (ruby-bdb1-0.2.1)
---> Packages processed: 3 done, 0 ignored, 0 skipped and 0 failed
---> Session ended at: Thu, 23 Dec 2004 20:35:33 -0500 (consumed 00:13:21)
freebsd521# portupgrade -vf lang/ruby18
---> Session started at: Thu, 23 Dec 2004 21:41:17 -0500
---> Reinstallation of lang/ruby18 started at: Thu, 23 Dec 2004 21:41:22 -0500
---> Reinstalling 'ruby-1.8.2.p4' (lang/ruby18)
---> Build of lang/ruby18 started at: Thu, 23 Dec 2004 21:41:22 -0500
---> Building '/usr/ports/lang/ruby18'
===> Cleaning for ruby-1.8.2.p4
===> Vulnerability check disabled
===> Extracting for ruby-1.8.2.p4
=> Checksum OK for ruby/ruby-1.8.2-preview4.tar.gz.
...edited...
---> Installation of lang/ruby18 ended at: Thu, 23 Dec 2004 22:01:08 -0500 (consumed 00:07:04)
---> Cleaning out obsolete shared libraries
[Updating the pkgdb <format:bdb1_btree> in /var/db/pkg ... - 12 packages found (-0 +1) . done]
---> Reinstallation of lang/ruby18 ended at: Thu, 23 Dec 2004 22:01:21 -0500 (consumed 00:19:59)
---> Listing the results (+:done / -:ignored / *:skipped / !:failed)
+ lang/ruby18 (ruby-1.8.2.p4)
---> Packages processed: 1 done, 0 ignored, 0 skipped and 0 failed
---> Session ended at: Thu, 23 Dec 2004 22:01:24 -0500 (consumed 00:20:07)
freebsd521# pkg_deinstall -vri lang/ruby16
...edited...
---> Listing the results (+:done / -:ignored / *:skipped / !:failed)
+ ruby16-shim-ruby18-1.8.1.p3
+ ruby-1.6.8.2004.07.28_1
---> Packages processed: 2 done, 0 ignored, 0 skipped and 0 failed
We have reached the end of the update process for Portupgrade and Ruby, advocated by /usr/ports/UPDATING. Remember that was an involved process because we were upgrading the very package-upgrade infrastructure itself.
The last invocation of Portupgrade was complicated. In this section I show the way Portupgrade is more commonly used.
The following represents a far more likely scenario: you run portversion(1) to discover what packages require updating. The -v switch means "verbose" and the -l "<" syntax tells portversion(1) to show only those packages needing to be updated:
freebsd521# portversion -v -l "<" bsdiff-4.1 < needs updating (port has 4.2) freebsd-update-1.4 < needs updating (port has 1.6_1)Here we see bsdiff(1) and freebsd-update(1) require updating.
The easiest way to proceed is to tell Portupgrade to upgrade all old packages. We invoke it with switches best described by Dru Lavigne: "the -R will check the build dependencies and the -r will check the applications that depend upon the port being upgraded. This will prevent your system from having outdated dependencies and software incompatibilities." The -v switch means be verbose and the -a says act on all packages needing to be upgraded:
freebsd521# portupgrade -varR
---> Session started at: Thu, 23 Dec 2004 22:24:42 -0500
** No need to upgrade 'ruby-1.8.2.p4' (>= ruby-1.8.2.p4). (specify -f to force)
** No need to upgrade 'cvsup-without-gui-16.1h' (>= cvsup-without-gui-16.1h). (specify -f to force)
** No need to upgrade 'mysql-client-3.23.58_3' (>= mysql-client-3.23.58_3). (specify -f to force)
---> Upgrade of misc/bsdiff started at: Thu, 23 Dec 2004 22:25:24 -0500
---> Upgrading 'bsdiff-4.1' to 'bsdiff-4.2' (misc/bsdiff)
---> Build of misc/bsdiff started at: Thu, 23 Dec 2004 22:25:24 -0500
---> Building '/usr/ports/misc/bsdiff'
===> Cleaning for bsdiff-4.2
===> Vulnerability check disabled
=> bsdiff-4.2.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
=> Attempting to fetch from http://www.daemonology.net/bsdiff/.
Receiving bsdiff-4.2.tar.gz (7686 bytes): 100%
7686 bytes transferred in 0.0 seconds (354.67 kBps)
===> Extracting for bsdiff-4.2
=> Checksum OK for bsdiff-4.2.tar.gz.
===> Patching for bsdiff-4.2
...edited...
---> Installation of security/freebsd-update ended at: Thu, 23 Dec 2004 22:26:53 -0500 (consumed 00:00:12)
---> Cleaning out obsolete shared libraries
[Updating the pkgdb <format:bdb1_btree> in /var/db/pkg ... - 10 packages found (-0 +1) . done]
---> Upgrade of security/freebsd-update ended at: Thu, 23 Dec 2004 22:26:59 -0500 (consumed 00:00:49)
** No need to upgrade 'perl-5.6.1_15' (>= perl-5.6.1_15). (specify -f to force)
---> Listing the results (+:done / -:ignored / *:skipped / !:failed)
- lang/ruby18 (ruby-1.8.2.p4)
- net/cvsup-without-gui (cvsup-without-gui-16.1h)
- databases/mysql323-client (mysql-client-3.23.58_3)
+ misc/bsdiff (bsdiff-4.1)
- databases/ruby-bdb1 (ruby18-bdb1-0.2.2)
- sysutils/portupgrade (portupgrade-20041224)
- shells/bash2 (bash-2.05b.007_2)
- misc/screen (screen-4.0.2_1)
+ security/freebsd-update (freebsd-update-1.4)
- lang/perl5 (perl-5.6.1_15)
---> Packages processed: 2 done, 8 ignored, 0 skipped and 0 failed
---> Session ended at: Thu, 23 Dec 2004 22:27:03 -0500 (consumed 00:02:21)
This process updated freebsd-update(1) and bsdiff(1).
A final re-run of portversion(1) shows all packages are up-to-date with the installed ports tree:
freebsd521# portversion -v -l "<" freebsd521#
Earlier I presented a way to keep the ports tree up-to-date using CVSup. I recently became a fan of a new tool, written by FreeBSD Update author Colin Percival. It's called Portsnap, and it resides it sysutils/portsnap. [4] Colin's documentation is thorough, so I will demonstrate a sample run.
Install Portsnap using your favorite mechanism. Be sure to make a copy of the /usr/local/etc/portsnap.conf file as described by the tool's installation process.
Once installed, run 'portsnap fetch':
freebsd521# portsnap fetch Fetching public key... done. Fetching snapshot tag... done. Fetching snapshot generated at Wed Dec 22 19:32:14 EST 2004: Receiving 7dd7712493ec82aa7214e257d5aa9a9f5129af0 (31939389 bytes): 5% 31939389 bytes transferred in 137.9 seconds (226.17 kBps) Extracting snapshot... done. Verifying snapshot integrity... done. Fetching updated snapshot tag... done. Updating from Wed Dec 22 19:32:14 EST 2004 to Thu Dec 23 22:31:53 EST 2004. Attempting to generate index via delta compression... success. Generating list of updates needed... 396 files or ports need to be updated. Attempting to fetch 370 patches... 370 fetched. Attempting to apply patches... done. Attempting to fetch 26 new files or ports... done.The first time 'portsnap fetch' is run, the process retrieves a 30 MB compressed snapshot of the ports tree. Now we extract it:
freebsd521# portsnap extract /usr/ports/.cvsignore /usr/ports/CHANGES /usr/ports/LEGAL /usr/ports/MOVED /usr/ports/Makefile /usr/ports/Mk/bsd.autotools.mk /usr/ports/Mk/bsd.emacs.mk /usr/ports/Mk/bsd.gnome.mk ...edited... /usr/ports/x11/yalias/ /usr/ports/x11/yelp/ /usr/ports/x11/zenity/Note: for future runs, you do not have to execute 'portsnap extract' again. Run 'portsnap update' instead. This will be demonstrated in the following section.
Next, execute 'make fetchindex' to download a new INDEX-5 and 'portsdb -u' to update the INDEX.db file:
freebsd521# cd /usr/ports freebsd521# make fetchindex Receiving INDEX-5.bz2 (612294 bytes): 100% 612294 bytes transferred in 2.4 seconds (250.72 kBps) freebsd521# ls -al /usr/ports/IND* -rw-r--r-- 1 root wheel 5970410 Dec 24 09:05 /usr/ports/INDEX-5 -rw-r--r-- 1 root wheel 2048 Dec 23 18:21 /usr/ports/INDEX.db freebsd521# portsdb -u [Updating the portsdb <format:bdb1_btree> in /usr/ports ... - 12148 port entries found .........1000.........2000.........3000.........4000.........5000.........6000.........7000... ......8000.........9000.........10000.........11000.........12000. ..... done] freebsd521# ls -al /usr/ports/IND* -rw-r--r-- 1 root wheel 5970410 Dec 24 09:05 /usr/ports/INDEX-5 -rw-r--r-- 1 root wheel 12295168 Dec 24 10:07 /usr/ports/INDEX.dbWhen done we run portversion(1) and find our packages are up-to-date:
freebsd521# portversion -v -l "<" [Updating the pkgdb <format:bdb1_btree> in /var/db/pkg ... - 12 packages found (-0 +2) .. done] freebsd521#
Now that we've set up Portsnap, let's invoke it again. Assume 24 hours have passed, so port maintainers have made updates to the ports tree. The process which follows is the one I use to keep my packages up-to-date.
I never run it as an automated process, say via cron(1). Why? First, it is important to check /usr/ports/UPDATING to see if any of your applications require special handling. Second, if the upgrade doesn't go smoothly, I prefer to be there to handle it.
In brief, the process is as follows:
cd /usr/ports portsnap fetch portsnap update make fetchindex portsdb -u portversion -v -l "<" Check /usr/ports/UPDATING for information relating to my applications portupgrade -varRHere is a sample run showing the www/sarg package requiring an upgrade:
freebsd521# cd /usr/ports freebsd521# portsnap fetch Fetching updated snapshot tag... done. Updating from Thu Dec 23 22:31:53 EST 2004 to Fri Dec 24 09:32:12 EST 2004. Attempting to generate index via delta compression... success. Generating list of updates needed... 47 files or ports need to be updated. Attempting to fetch 46 patches... 46 fetched. Attempting to apply patches... done. Attempting to fetch 1 new files or ports... done. freebsd521# portsnap update Removing old files and directories... done. Extracting new files: /usr/ports/Mk/bsd.emacs.mk /usr/ports/audio/sox/ /usr/ports/comms/obexapp/ /usr/ports/databases/fastdb/ ...edited... /usr/ports/x11-fonts/dejavu/ /usr/ports/x11-toolkits/open-motif/ freebsd521# portsdb -u [Updating the portsdb <format:bdb1_btree> in /usr/ports ... - 12148 port entries found .........1000.........2000.........3000.........4000.........5000.........6000.........7000... ......8000.........9000.........10000.........11000.........12000. ..... done] freebsd521# portversion -v -l "<" sarg-1.4.1 < needs updating (port has 2.0.2)Checking /usr/ports/UPDATING shows no issue for updating www/sarg.
freebsd521# portupgrade -varR
---> Session started at: Fri, 24 Dec 2004 10:17:52 -0500
---> Upgrade of www/sarg started at: Fri, 24 Dec 2004 10:18:00 -0500
---> Upgrading 'sarg-1.4.1' to 'sarg-2.0.2' (www/sarg)
---> Build of www/sarg started at: Fri, 24 Dec 2004 10:18:00 -0500
---> Building '/usr/ports/www/sarg'
===> Cleaning for sarg-2.0.2
===> Vulnerability check disabled
=> sarg-2.0.2.tar.gz doesn't seem to exist in /usr/ports/distfiles/.
...edited...
---> Installation of www/sarg ended at: Fri, 24 Dec 2004 10:24:37 -0500 (consumed 00:00:13)
---> Cleaning out obsolete shared libraries
[Updating the pkgdb <format:bdb1_btree> in /var/db/pkg ... - 13 packages found (-0 +1) . done]
---> Upgrade of www/sarg ended at: Fri, 24 Dec 2004 10:24:43 -0500 (consumed 00:06:43)
** No need to upgrade 'freebsd-sha1-5.3' (>= freebsd-sha1-5.3). (specify -f to force)
** No need to upgrade 'perl-5.6.1_15' (>= perl-5.6.1_15). (specify -f to force)
---> Listing the results (+:done / -:ignored / *:skipped / !:failed)
+ www/sarg (sarg-1.4.1)
- sysutils/freebsd-sha1 (freebsd-sha1-5.3)
- lang/ruby18 (ruby-1.8.2.p4)
- net/cvsup-without-gui (cvsup-without-gui-16.1h)
- misc/bsdiff (bsdiff-4.2)
- security/freebsd-update (freebsd-update-1.6_1)
- sysutils/portsnap (portsnap-0.3.1)
- databases/mysql323-client (mysql-client-3.23.58_3)
- databases/ruby-bdb1 (ruby18-bdb1-0.2.2)
- sysutils/portupgrade (portupgrade-20041224)
- shells/bash2 (bash-2.05b.007_2)
- misc/screen (screen-4.0.2_1)
- lang/perl5 (perl-5.6.1_15)
---> Packages processed: 1 done, 12 ignored, 0 skipped and 0 failed
---> Session ended at: Fri, 24 Dec 2004 10:25:03 -0500 (consumed 00:07:10)
That's it -- everything needing an upgrade has been upgraded.
Imagine you run FreeBSD on a slow laptop (named "orr") and on a relatively fast server (named "janney"). It doesn't make sense to let the laptop labor while it builds packages if you could build them on the server. The following explains how I often build packages on a server and then install them on my laptop.
First, on the server, take the steps indicated earlier to update the ports tree, INDEX-5, and INDEX.db. You run the portversion(1) command and you see that three packages need to be upgraded:
janney:/usr/ports# portversion -v -l "<" bash-3.0.15 < needs updating (port has 3.0.16_1) freebsd-update-1.6 < needs updating (port has 1.6_1) sudo-1.6.8.1 < needs updating (port has 1.6.8.4)You use all three on the server and the laptop. After reading /usr/ports/UPDATING, you invoke Portupgrade with an extra switch: -p
janney:/usr/ports# portupgrade -varRpThe -p switch tells Portupgrade to build packages for the applications it upgrades.
When done I had three new packages in /usr/ports/packages/All. This is where packages built during the upgrade or installation process are stored:
janney:/usr/ports/packages/All# ls -alt total 135276 drwxr-xr-x 21 root wheel 512 Nov 25 11:03 .. -rw-r--r-- 1 root wheel 545101 Nov 25 11:03 bash-3.0.16_1.tbz drwxr-xr-x 2 root wheel 1536 Nov 25 11:03 . -rw-r--r-- 1 root wheel 10097 Nov 25 10:59 portsnap-0.2_1.tbz -rw-r--r-- 1 root wheel 29219 Nov 25 10:58 freebsd-update-1.6_1.tbz -rw-r--r-- 1 root wheel 99157 Nov 25 10:58 sudo-1.6.8.4.tbzI expected bash, freebsd-update, and sudo to be there. These three were the packages identified by portversion(1) as being out-of-date. A new portsnap package was created as part of the upgrade process for freebsd-update, since portsnap depends upon freebsd-update to function.
Now that I had these new packages, I turned to updating my laptop, named "orr." I followed the five steps I first did for janney, and a 'portversion -v -l "<"' to see what needed updating. Laptop orr has a lot more packages installed compared to janney. If I want to avoid updating packages on orr via building from source through the ports tree, I need to obtain updated packages either from the FreeBSD project or from a system that's built the same packages.
Standard FreeBSD systems do not seem to have a means to build packages without installing them. (I believe OpenBSD has this capabilitity.) Since I prefer to not maintain a "package builder" with every application I need, I take a dual-pronged approach. First, I wait until the FreeBSD project provides updated packages. Second, for critical applications that tend to be installed on many systems, I create my own packages. This second approach is what this process describes.
Based on my update on server janney, I know I can provide updated bash, sudo, and freebsd-update packages for laptop orr. To do that I NFS mount /usr/ports/packages/All on janney to orr, then run portupgrade(1). I use the -PP switch to tell portupgrade(1) to only use packages to update applications:
orr:/usr/ports# mount -t nfs janney:/usr/ports/packages/All /usr/ports/packages/All orr:/usr/ports# portupgrade -varRPP ---> Session started at: Thu, 25 Nov 2004 11:26:02 -0500 ** No need to upgrade 'tcpflow-0.21' (>= tcpflow-0.21). (specify -f to force) ---> Checking for the latest package of 'security/sudo' ---> Found a package of 'security/sudo': sudo-1.6.8.4.tbz (sudo-1.6.8.4) ---> Upgrade of security/sudo started at: Thu, 25 Nov 2004 11:26:09 -0500 ---> Upgrading 'sudo-1.6.8.1' to 'sudo-1.6.8.4' (security/sudo) using a package ---> Updating dependency info ---> Uninstallation of sudo-1.6.8.1 started at: Thu, 25 Nov 2004 11:26:11 -0500 ---> Fixing up dependencies before creating a package ---> Backing up the old version ---> Uninstalling the old version ---> Deinstalling 'sudo-1.6.8.1' [Updating the pkgdb in /var/db/pkg ... - 167 packages found (-1 +0) (...) done] ---> Uninstallation of sudo-1.6.8.1 ended at: Thu, 25 Nov 2004 11:26:15 -0500 (consumed 00:00:03) pkg_info: can't find package 'sudo-1.6.8.4.tbz' installed or in a file! ---> Installation of sudo-1.6.8.4 started at: Thu, 25 Nov 2004 11:26:15 -0500 ---> Installing the new version via the package Will not overwrite existing /usr/local/etc/sudoers file. ---> Removing temporary backup files ---> Installation of sudo-1.6.8.4 ended at: Thu, 25 Nov 2004 11:26:18 -0500 (consumed 00:00:03) ---> Cleaning out obsolete shared libraries [Updating the pkgdb in /var/db/pkg ... - 168 packages found (-0 +1) . done] ---> Upgrade of security/sudo ended at: Thu, 25 Nov 2004 11:26:20 -0500 (consumed 00:00:10) ...truncated... orr:/usr/ports# umount /usr/ports/packages/AllWhen done, bash, freebsd-update, and sudo were updated.
Notice that I did not have to run Portsnap, or 'make index', or portsdb(1) on the laptop prior to running portupgrade(1). Because I had mounted janney's /usr/ports over orr's /usr/ports, my upgrade process used the ports tree, INDEX-5, and INDEX.db files on the server.
So far I've described ways to update packages without explaining why an administrator might want to take that action. The first reason is to address shortcomings in the older version of an application. A second reason is to provide additional features needed by users. A third, and potentially most important, reason is to fix security problems.
FreeBSD's Portaudit tool (found in security/portaudit) checks the security status of packages. Portaudit uses the Vulnerability and eXposure Markup Language, "an XML application for documenting security issues in a software package collection" like the FreeBSD ports system. [5] You can browse the FreeBSD or OpenBSD VuXML pages to see vulnerabilities recorded since the VuXML project began in late 2003.
Using the VuXML database is as simple as installing the Portaudit port. Be sure to begin with an up-to-date ports tree. Install Portaudit, and then run it as shown to check installed packages for problems. The -F flag tells Portaudit to fetch a new copy of the vulnerability database, while -d says show the database creation time and -a says check all installed ports/packages.
freebsd521# portaudit -Fda Receiving auditfile.tbz (17217 bytes): 100% 17217 bytes transferred in 0.2 seconds (86.66 kBps) New database installed. Database created: Fri Dec 24 10:40:15 EST 2004 Affected package: mysql-client-3.23.58_3 Type of problem: mysql -- mysql_real_connect buffer overflow vulnerability. Reference: <http://www.FreeBSD.org/ports/portaudit/835256b8-46ed-11d9-8ce0-00065be4b5b6.html> Affected package: FreeBSD-502010 Type of problem: multiple vulnerabilities in the cvs server code. Reference: <http://www.FreeBSD.org/ports/portaudit/d2102505-f03d-11d8-81b0-000347a4fa7d.html> Note: To disable this check add the uuid to `portaudit_fixed' in /usr/local/etc/portaudit.conf 1 problem(s) in your installed packages found. You are advised to update or deinstall the affected package(s) immediately.Since Portaudit found problems in mysql-client-3.23.58_3, we should upgrade that port immediately. Since a newer version is not available, we might have to upgrade to the 4.0 or 4.1 MySQL client.
Portaudit also reported a problem with the CVS server code. However, we know this is fixed because we use FreeBSD Update to keep the OS up-to-date:
freebsd521# freebsd-update fetch Fetching updates signature... Fetching hash list signature... Examining local system... No updates availableWe take the advice output by Portaudit and add this line to /usr/local/etc/portaudit.conf:
# this vulnerability has been fixed in your FreeBSD version portaudit_fixed="d2102505-f03d-11d8-81b0-000347a4fa7d"We re-run Portaudit after making this entry and removing the MySQL client:
freebsd521# portaudit -Fda New database installed. Database created: Fri Dec 24 10:40:15 EST 2004 0 problem(s) in your installed packages found.Portaudit works with sysutils/pkg_install-devel to warn sys admins when they try to install vulnerable software. In the following example, I try to install Ethereal using an out-of-date ports tree. The Ethereal port wants to install version 0.10.0a, which has multiple problems.
janney:/usr/ports/net/ethereal# make ===> ethereal-0.10.0a_2 has known vulnerabilities: >> multiple vulnerabilities in ethereal. Reference: cdf18ed9-7f4a-11d8-9645-0020ed76ef5a.html> >> Please update your ports tree and try again. *** Error code 1 Stop in /usr/ports/net/ethereal.If you are willing to accept the risks of a vulnerable application, you can disable the vulnerability checking manually. In the following example, another MySQL client has a problem:
neely:/usr/ports/databases/mysql40-client$ make ===> mysql-client-4.0.18_1 has known vulnerabilities: >> MySQL insecure temporary file creation (mysqlbug). Reference: >> Please update your ports tree and try again.This is a minor problem affecting only the 'mysqlbug' script, not core MySQL client functionality. We may not see a fix in the MySQL distribution until 4.0.19. Thanks to Michael Nottebrock, I learned how to install a port with a vulnerability:
neely:/usr/ports/databases/mysql40-client$ make -DDISABLE_VULNERABILITIES ===> Vulnerability check disabled >> mysql-4.0.18.tar.gz doesn't seem to exist in /usr/ports/distfiles/. ...truncated...The package is now installed.
Use make -DDISABLE_VULNERABILITIES with care!
Portaudit can be used to check the status of a port before it is installed. Here we check for vulnerabilities in the Racoon port. By passing Portaudit the -C flag, we tell it to compare that specific port with the VuXML database.
janney:/usr/ports/security/racoon# portaudit -C Port racoon-20040116a (security/racoon) should be marked FORBIDDEN: - http://people.freebsd.org/~eik/portaudit/ccd698df-8e20-11d8-90d1-0020ed76ef5a.html - http://people.freebsd.org/~eik/portaudit/40fcf20f-8891-11d8-90d1-0020ed76ef5a.html - http://people.freebsd.org/~eik/portaudit/d8769838-8814-11d8-90d1-0020ed76ef5a.html - http://people.freebsd.org/~eik/portaudit/f8551668-de09-4d7b-9720-f1360929df07.htmlIf we ran 'portaudit -A' in the /usr/ports directory, Portaudit would check for vulnerabilities in the entire ports tree.
Portaudit integrates with the scripts run on a daily basis to notify system administrators of security problems. For example, here is output from the "security run output" from one of my systems:
Checking for packages with security vulnerabilities: Affected package: ruby-1.8.1_2 Type of problem: ruby -- CGI DoS. Reference: <http://people.freebsd.org/~eik/portaudit/d656296b-33ff-11d9-a9e7-0001020eed82.html> Affected package: mutt-1.4.1_4 Type of problem: Buffer overflow in Mutt 1.4. Reference: <http://people.freebsd.org/~eik/portaudit/67c05283-5d62-11d8-80e3-0020ed76ef5a.html> 2 problem(s) in your installed packages found. You are advised to update or deinstall the affected package(s) immediately.This is an excellent feature that makes system administration much easier.
I hope this article has helped you understand the different ways to keep FreeBSD applications up-to-date. It is by no means comprehensive, but by following it you hopefully can judge the different ways to keep your applications current.
Acknowledgements
As readers provide feedback, I will credit them here.
References
Revision History