Tag Archives: freebsd

Update FreeBSD to 14.0, Update zpool, and can’t boot…how to fix?

As FreeBSD 13.2 is EOF and FreeBSD 14.0 is released, so I think it’s time to update my server to FreeBSD 14.

So I updated the server as what I did before.

freebsd-update -r 14.0-RELEASE upgrade
freebsd-update install
shutdown -r now
freebsd-update install

And when I also noticed that FreeBSD 14 updated zpool verson and introduce some new features, so I also updated zpool as well.

zpool upgrade zroot

And also updated bootcode on all disks, this is very important.

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da0
...
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 da7

And the final step is to reboot the server to finish upgrade.

Then, server can’t boot. I got below error.

And gpart can find all disks and partition.

What’s happening? I updated bootcode why I still get this error?

And then I did some research, and found that in FreeBSD 14 release note, it’s mentioned

There have been a number of improvements in the boot loaders, and upgrading the boot loader on the boot partition is recommended in most cases, in particular if the system boots via EFI. If the root is on a ZFS file system, updating the boot loader is mandatory if the pool is to be upgraded, and the boot loader update must be done first. Note that ZFS pool upgrades are not recommended for root file systems in most cases, but updating the boot loader can avoid making the system unbootable if the pool is upgraded in the future. The bootstrap update procedure depends on the boot method (EFI or BIOS), and also on the disk partitioning scheme. The next several sections address each in turn.

Well, this is something new in FreeBSD 14, and I didn’t update EFI boot file and that’s why I’m facing this issue.

So, spend some time to fix the boot issue.

First, I tried to change bios from BIOS to UEFI, to see if I can boot the server via BIOS. But unfortunately, it’s not working, and got below error.

So, the only choice is to boot the server with liveCD and fix it.

Select 1, boot installer here. Do not select 2 single mode as single mode is readonly.

Select liveCD

Type root to login, no password is needed.

And then you can mount your zpool via below command (Actually this is not needed if you only need to fix the boot issue)

zpool import -f -R /tmp/zroot zroot

Keep in mind here the only writable folder is /tmp, so you must mount your zpool here.

Determined the location of the boot loader via below command:

efibootmgr -v.

The one with + is the one in use, in below screen shot as I boot from virtual CD so it’s marked as USB.

The one on local disk is mfisyspd0p1

Mount the ESP partition via below command:

mount_msdosfs /dev/mfisyspd0p1 /boot/efi

And then run below files to copy the EFI boot loaders from CD to local disk.


cp /boot/loader.efi /mnt/efi/freebsd/loader.efi
cp /boot/loader.efi /mnt/efi/boot/bootx86.efi

Then reboot,

Then I got an error complaint UEFI is broken.

Then tried to boot into livecd and copy the files again, then reboot.

This time no error, just complaint not able to boot.

So what’s happening?

I remember when I copied the EFI loader, there are some errors from the HBA card.

Then I checked the sha256 of the loader.efi on CD and on local disk, looks like the loader on local disk is corrupted.

This is a known issue for several years. The default drive in FreeBSD is not working properly with mfi driver (I’m using dell h330). So the fix is

Option 1, when you use livecd to boot, select 3 and enter below command to load the right mrsas driver:

set hw.mfi.mrsas_enable="1"

boot

 

Option 2, keeps copy the file to hard disk, and then use command sync to force OS to write data to disk.  And then check the sha256 of the file. Due to the default driver is not stable, I tried about 10 times, finally got the loader.efi copied to the right position.

Then reboot. This time server can boot successfully, but it booted into single mode.

That’s because we replace the file via livecd, so just run fsck to fix it

fsck /dev/da0p1

Then reboot, all good now.

 

Increasing disk/zpool size of FreeBSDZFS disks in Linode

I’m running a FreeBSD instance in Linode. And FreeBSD is not the official supported OS in Linode.The disk type is set to raw to install FreeBSD.

And recently Linode upgraded my instance and the disk size is increased from 40G to 80G.

But when I login to the system, I found that my zpool is still 40G. But the disk is shown as 80G.

# gpart show
=>       34  100663229  ada0  GPT  (80G) [CORRUPT]
         34          6        - free -  (3.0K)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048    4194304     2  freebsd-swap  (2.0G)
    4196352   96464896     3  freebsd-zfs  (46G)
  100661248       2015        - free -  (1.0M)

I tried to enable zfs autoexpand on my zpool, the same. Then how to increase the disk/zpool size?

First, re-write disk metadata

# gpart recover ada0
ada0 recovered

After this, gpart can show the real disk space info

#:~ # gpart show
=>       40  167772080  ada0  GPT  (80G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048    4194304     2  freebsd-swap  (2.0G)
    4196352   96464896     3  freebsd-zfs  (46G)
  100661248   67110872        - free -  (32G)

Then, expand zfs partition

# gpart resize -i 3 ada0
ada0p3 resized

Then, expand the zpool

zpool online -e zroot /dev/ada0p3

Done!

#:~ # df -h /
Filesystem            Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default     74G     29G     45G    39%    /

Don’t forget to write zfs info to disk

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

FreeBSD + Nginx : Enable HTTP/2 and ALPN

For now more and more servers are starting using HTTP/2 which is faster and more secure.
This post is about how to enable HTTP/2 on FreeBSD servers.

Nginx Stable 1.8.* doesn’t support HTTP/2. So we need to install nginx-devel (version 1.9.*) first. If you have already installed Nginx stable, you need to uninstall it first.

And before you install nginx-devel, you need to install openssl from port first. Otherwise nginx will use system based openssl library, and you can’t enable ALPN for http/2. That’s because ALPN requires openssl 1.0.2*, and the system based openssl is version 0.98

So the first step is:

cd /usr/ports/security/openssl
make install

And then add below line into your make.conf to make sure that you’ll use the latest openssl library to build nginx.

cd /usr/ports/security/openssl
WITH_OpeNSSL_PORTS=yes

Then you can install nginx-devel

cd /usr/ports/www/nginx-devel
make config

Make sure that you select HTTP_SSL and HTTPV2. Please be aware that SPDY is no longer supported by nginx 1.9.*.
Then install it.

make install

Go back to your nginx.conf, and modify it as following:

  
server {
                listen       443  accept_filter=dataready ssl http2;
                ssl on;
                ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
                ssl_certificate /usr/local/etc/nginx/yourcert.crt;
                ssl_certificate_key /usr/local/etc/nginx/yourcert.key;
                ssl_ciphers EECDH+CHACHA20:EECDH+CHACHA20-draft:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
                ssl_prefer_server_ciphers on;
                ssl_session_cache shared:SSL:50m;
                ssl_session_timeout 1d;
                ssl_session_tickets on;
                ssl_stapling on;
                resolver 8.8.8.8 8.8.4.4 valid=300s;
                resolver_timeout 10s;
                ssl_trusted_certificate /usr/local/etc/nginx/ca-certs.crt;
                ssl_stapling_verify on;
                add_header Strict-Transport-Security "max-age=15552000; includeSubdomains; preload";
                ....
        }

I’d like to explain a little bit for this configuration.

  
listen       443  accept_filter=dataready ssl http2;

This is about to enable HTTP/2 and SSL.
You may notice that here I’m using accept_filter=dataready instead of accept_filter=httpready. There are currently two filters in FreeBSD: “dataready” and “httpready” which need to started at boot by adding accf_data_load=”YES” and accf_http_load=”YES” to /boot/loader.conf. dataready waits for the first properly formed packet to arrive from the client before passing the request to nginx. httpready waits not only for the packets, but also for the end of the HTTP header before passing the request onto nginx. Keep in mind “httpready” filter breaks support for ancient HTTP/0.9 because v0.9 does not have any headers. HTTP/0.9 is so old we are not going to worry about support it and since a HTTP/0.9 would not have the newer SSL ciphers anyways.

To configure nginx to use the accept filters in FreeBSD we need to add the arguments to the listen directive. Since http (port 80) is unencrypted we can use the “accept_filter=httpready” accept filter. This is because FreeBSD will need to look at the packet and parse the complete http header. SSL (https port 443) is encrypted so FreeBSD can not parse the packets so we need to use the “accept_filter=dataready” accept filter. Both accept filter examples can be found in the configuration below. To use FreeBSD accept filters you must enable them in /boot/loader.conf to load on boot.

And someone may use the nginx 1.9.* new feature reuseport. I have to say, unfortunately, this new feature doesn’t support FreeBSD. I made several test on my server, it seems if you enable “reuseport” on your server, then all your traffic will be handled by your first worker! The OS can’t balance the workers’ workload. It means if your server is very busy, with reuseport on FreeBSD will significantly slow down your server. You’ll find that your first worker is taking up 100% CPU and the rest are idle! So, at this moment, do not enable “reuseport” on FreeBSD Nginx.

  
                ssl on;
                ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
                ssl_certificate /usr/local/etc/nginx/yourcert.crt;
                ssl_certificate_key /usr/local/etc/nginx/yourcert.key;

This part is just enable SSL. It’s pretty easy. There is only one thing you need to take care, that is for yourcert.crt, there is no need to put Root certificate into it. Just put your server certificate and intermediate certificate into it to reduce the size of your certificate to reduce the connection time.

  
                ssl_ciphers EECDH+CHACHA20:EECDH+CHACHA20-draft:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
                ssl_prefer_server_ciphers on;

These SSL ciphers are recommended by cloudflare. It can support most browsers. But IE6 is not supported.

  
                ssl_session_cache shared:SSL:5m;
                ssl_session_timeout 180m;
                ssl_session_tickets on;

NGINX caches the session parameters used to create the SSL/TLS connection. This cache, shared among all workers when you include the shared parameter, drastically improves response time for subsequent requests because the connection setup information is already known. Assign a name to the cache and set its size (a 1-MB shared cache accommodates approximately 4,000 sessions).The ssl_session_timeout directive controls how long the session information remains in the cache. The default value is 5 minutes; increasing it to several hours (as in the following example) improves performance but requires a larger cache. Session tickets store information about specific SSL/TLS sessions. When a client resumes interaction with an application, the session ticket is used to resume the session without renegotiation. Session IDs are an alternative; an MD5 hash is used to map to a specific session stored in the cache created by the ssl_session_cache directive.

  
                ssl_stapling on;
                resolver 8.8.8.8 8.8.4.4 valid=300s;
                resolver_timeout 10s;
                ssl_trusted_certificate /usr/local/etc/nginx/ca-certs.crt;
                ssl_stapling_verify on;

OCSP stapling, can decreases the time of the SSL/TLS handshake. Traditionally, when a user connects to your application or website via HTTPS, his or her browser validates the SSL certificate against a certificate revocation list (CRL) or uses an Online Certificate Status Protocol (OCSP) record from a certificate authority (CA). These requests add latency and the CAs can be unreliable. With NGINX you can cache the OCSP response to your server and eliminate costly overhead.

At this step you need to create your ssl_trusted_certificate. Please note that for OCSP certificate (ssl_trusted_certificate /usr/local/etc/nginx/ca-certs.crt), you need to put your root certificate and intermediate certificate into it, and no server certificate. And the right order is root certificate first, then intermediate certificate, then second intermediate certificate.

  
                add_header Strict-Transport-Security "max-age=15552000; includeSubdomains; preload";

Enable HSTS for your server. Please be aware that if you added this into your configure, then you can’t remove it and go back to http again. Otherwise end users can’t access your website because they will always redirect to https site.

Now, you can go to http://www.ssllabs.com to test your website. You should be able to get score A+

Install FreeBSD on Linode VPS

I’m a FreeBSD fan, I prefer to use FreeBSD as the web server if possible.
In the pass Linode doesn’t support FreeBSD, people who want to use FreeBSD have to choose Digital Ocean or other service provider.
And in earlier this year, Linode has migrated all VPS from Xen to KVM, and it made installing FreeBSD on Linode VPS possible. But FreeBSD is not official supported by Linode. You can’t deploy FreeBSD via Linode control panel. So I wrote this article to share my experience about how to install FreeBSD on Linode.

1. Add a new linode VPS.

1

2. Go to the dashboard, and create a new vDisk.

2

Continue reading

use unix socket instead of tcp connection for memcached to speed up web servers

By default, memcached is listening on port 11211 for all connections. But for a single server, it’s better to config memcached to use unix socket which improve the performance and reduce the legacy.

On freebsd, it’s easy to change from TCP/IP to unix socket. But there are some steps you need to pay more attention.

1. Modify rc.conf to force memcached to use unix socket.

memcached_flags="-m 2048 -s /tmp/memcached.sock"

2. Modify the connection string in your php files.
For TCP connection, use

$_config['memory']['memcache']['server'] = '127.0.0.1';
$_config['memory']['memcache']['port'] = 11211;

If you are using php-memcache, you should change it to

$_config['memory']['memcache']['server'] = 'unix:///tmp/memcached.sock';
$_config['memory']['memcache']['port'] = 0;

If you are using php-memcached, you should change it to

$_config['memory']['memcache']['server'] = '/tmp/memcached.sock';
$_config['memory']['memcache']['port'] = 0;

3. Another thing I found in Freebsd is that if you force memcached to use unix socket, every time you restart the service, memcached will change the socket file permission to 550 which means php can’t connect to the socket file. So you need to grant php user access to that file.

chmod 777 /tmp/memcached.sock

You need to do this each time after you restart memcached.