Category Archives: FreeBSD

expand AWS EC2 FreeBSD ZFS disk

For testing purpose, I setup a Freebsd instance on AWS, which is using zfs on root.
And then I add 10G disk space to the root volume. Even though I enabled auto-expand for zroot, it seems the 10G space is not added to system. Here are the steps to expand the disk for zroot:

1. Reboot the server.  Even though it’s said that reboot is not necessary, I suggest to reboot the server to make sure server can recognize the new size.

2. As the disk size change, we need to fix the GPT partition table first.

gpart recovery ada0

3. Some documents said can use “zpool online -e” to expand the disk. As shown in above picture, the command can’t auto update GPT and assign the space to zfs partition.

4. We need to use gpart to update GPT first, then expand the zfs partition.

#gpart resize -i 2 ada0
#zpool online -e zrrot /dev/adap2

Increasing disk/zpool size of FreeBSDZFS disks in Linode

I’m running a FreeBSD instance in Linode. And FreeBSD is not the official supported OS in Linode.The disk type is set to raw to install FreeBSD.

And recently Linode upgraded my instance and the disk size is increased from 40G to 80G.

But when I login to the system, I found that my zpool is still 40G. But the disk is shown as 80G.

# gpart show
=>       34  100663229  ada0  GPT  (80G) [CORRUPT]
         34          6        - free -  (3.0K)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048    4194304     2  freebsd-swap  (2.0G)
    4196352   96464896     3  freebsd-zfs  (46G)
  100661248       2015        - free -  (1.0M)

I tried to enable zfs autoexpand on my zpool, the same. Then how to increase the disk/zpool size?

First, re-write disk metadata

# gpart recover ada0
ada0 recovered

After this, gpart can show the real disk space info

#:~ # gpart show
=>       40  167772080  ada0  GPT  (80G)
         40       1024     1  freebsd-boot  (512K)
       1064        984        - free -  (492K)
       2048    4194304     2  freebsd-swap  (2.0G)
    4196352   96464896     3  freebsd-zfs  (46G)
  100661248   67110872        - free -  (32G)

Then, expand zfs partition

# gpart resize -i 3 ada0
ada0p3 resized

Then, expand the zpool

zpool online -e zroot /dev/ada0p3

Done!

#:~ # df -h /
Filesystem            Size    Used   Avail Capacity  Mounted on
zroot/ROOT/default     74G     29G     45G    39%    /

Don’t forget to write zfs info to disk

gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

FreeBSD: use mrsas driver to replace mfi driver

My server got a Dell PERC H330 raid card, and I made it’s working in HBA mode to make sure that it can get best performance under FreeBSD with zfs.
But every time when I boot the server, I’ll get below error message and it take a long time to pass the disk check stage.

1

After research, it seems this timeout error was caused by the old mfi driver.
And LSI has released a new mrsas driver for FreeBSD. So It’s better to switch the driver from mfi to mrsas.

First, add below line into /boot/loader.conf

mrsas_load="YES"

And then add below device hint into /boot/device.hints . This line is very important. Without this device hint, FreeBSD will use the old mfi driver for raid card even though you enabled maras drvier.

hw.mfi.mrsas_enable="1"

And then, add below line into /boot/loader.conf to disable disk id identity.

kern.geom.label.gptid.enable="0"
kern.geom.label.disk_ident.enable="0"

Without above two line, after you switch from mfi to mrsas, all the disks will be shown as diskid-*****************。

And don’t forget to update /etc/fstab to change the swap partition from mfi*p* to da*p*. Otherwise you’ll lose your swap partition.

Then, reboot your server, and enjoy.

FreeBSD + Nginx : Enable HTTP/2 and ALPN

For now more and more servers are starting using HTTP/2 which is faster and more secure.
This post is about how to enable HTTP/2 on FreeBSD servers.

Nginx Stable 1.8.* doesn’t support HTTP/2. So we need to install nginx-devel (version 1.9.*) first. If you have already installed Nginx stable, you need to uninstall it first.

And before you install nginx-devel, you need to install openssl from port first. Otherwise nginx will use system based openssl library, and you can’t enable ALPN for http/2. That’s because ALPN requires openssl 1.0.2*, and the system based openssl is version 0.98

So the first step is:

cd /usr/ports/security/openssl
make install

And then add below line into your make.conf to make sure that you’ll use the latest openssl library to build nginx.

cd /usr/ports/security/openssl
WITH_OpeNSSL_PORTS=yes

Then you can install nginx-devel

cd /usr/ports/www/nginx-devel
make config

Make sure that you select HTTP_SSL and HTTPV2. Please be aware that SPDY is no longer supported by nginx 1.9.*.
Then install it.

make install

Go back to your nginx.conf, and modify it as following:

  
server {
                listen       443  accept_filter=dataready ssl http2;
                ssl on;
                ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
                ssl_certificate /usr/local/etc/nginx/yourcert.crt;
                ssl_certificate_key /usr/local/etc/nginx/yourcert.key;
                ssl_ciphers EECDH+CHACHA20:EECDH+CHACHA20-draft:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
                ssl_prefer_server_ciphers on;
                ssl_session_cache shared:SSL:50m;
                ssl_session_timeout 1d;
                ssl_session_tickets on;
                ssl_stapling on;
                resolver 8.8.8.8 8.8.4.4 valid=300s;
                resolver_timeout 10s;
                ssl_trusted_certificate /usr/local/etc/nginx/ca-certs.crt;
                ssl_stapling_verify on;
                add_header Strict-Transport-Security "max-age=15552000; includeSubdomains; preload";
                ....
        }

I’d like to explain a little bit for this configuration.

  
listen       443  accept_filter=dataready ssl http2;

This is about to enable HTTP/2 and SSL.
You may notice that here I’m using accept_filter=dataready instead of accept_filter=httpready. There are currently two filters in FreeBSD: “dataready” and “httpready” which need to started at boot by adding accf_data_load=”YES” and accf_http_load=”YES” to /boot/loader.conf. dataready waits for the first properly formed packet to arrive from the client before passing the request to nginx. httpready waits not only for the packets, but also for the end of the HTTP header before passing the request onto nginx. Keep in mind “httpready” filter breaks support for ancient HTTP/0.9 because v0.9 does not have any headers. HTTP/0.9 is so old we are not going to worry about support it and since a HTTP/0.9 would not have the newer SSL ciphers anyways.

To configure nginx to use the accept filters in FreeBSD we need to add the arguments to the listen directive. Since http (port 80) is unencrypted we can use the “accept_filter=httpready” accept filter. This is because FreeBSD will need to look at the packet and parse the complete http header. SSL (https port 443) is encrypted so FreeBSD can not parse the packets so we need to use the “accept_filter=dataready” accept filter. Both accept filter examples can be found in the configuration below. To use FreeBSD accept filters you must enable them in /boot/loader.conf to load on boot.

And someone may use the nginx 1.9.* new feature reuseport. I have to say, unfortunately, this new feature doesn’t support FreeBSD. I made several test on my server, it seems if you enable “reuseport” on your server, then all your traffic will be handled by your first worker! The OS can’t balance the workers’ workload. It means if your server is very busy, with reuseport on FreeBSD will significantly slow down your server. You’ll find that your first worker is taking up 100% CPU and the rest are idle! So, at this moment, do not enable “reuseport” on FreeBSD Nginx.

  
                ssl on;
                ssl_protocols  TLSv1 TLSv1.1 TLSv1.2;
                ssl_certificate /usr/local/etc/nginx/yourcert.crt;
                ssl_certificate_key /usr/local/etc/nginx/yourcert.key;

This part is just enable SSL. It’s pretty easy. There is only one thing you need to take care, that is for yourcert.crt, there is no need to put Root certificate into it. Just put your server certificate and intermediate certificate into it to reduce the size of your certificate to reduce the connection time.

  
                ssl_ciphers EECDH+CHACHA20:EECDH+CHACHA20-draft:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
                ssl_prefer_server_ciphers on;

These SSL ciphers are recommended by cloudflare. It can support most browsers. But IE6 is not supported.

  
                ssl_session_cache shared:SSL:5m;
                ssl_session_timeout 180m;
                ssl_session_tickets on;

NGINX caches the session parameters used to create the SSL/TLS connection. This cache, shared among all workers when you include the shared parameter, drastically improves response time for subsequent requests because the connection setup information is already known. Assign a name to the cache and set its size (a 1-MB shared cache accommodates approximately 4,000 sessions).The ssl_session_timeout directive controls how long the session information remains in the cache. The default value is 5 minutes; increasing it to several hours (as in the following example) improves performance but requires a larger cache. Session tickets store information about specific SSL/TLS sessions. When a client resumes interaction with an application, the session ticket is used to resume the session without renegotiation. Session IDs are an alternative; an MD5 hash is used to map to a specific session stored in the cache created by the ssl_session_cache directive.

  
                ssl_stapling on;
                resolver 8.8.8.8 8.8.4.4 valid=300s;
                resolver_timeout 10s;
                ssl_trusted_certificate /usr/local/etc/nginx/ca-certs.crt;
                ssl_stapling_verify on;

OCSP stapling, can decreases the time of the SSL/TLS handshake. Traditionally, when a user connects to your application or website via HTTPS, his or her browser validates the SSL certificate against a certificate revocation list (CRL) or uses an Online Certificate Status Protocol (OCSP) record from a certificate authority (CA). These requests add latency and the CAs can be unreliable. With NGINX you can cache the OCSP response to your server and eliminate costly overhead.

At this step you need to create your ssl_trusted_certificate. Please note that for OCSP certificate (ssl_trusted_certificate /usr/local/etc/nginx/ca-certs.crt), you need to put your root certificate and intermediate certificate into it, and no server certificate. And the right order is root certificate first, then intermediate certificate, then second intermediate certificate.

  
                add_header Strict-Transport-Security "max-age=15552000; includeSubdomains; preload";

Enable HSTS for your server. Please be aware that if you added this into your configure, then you can’t remove it and go back to http again. Otherwise end users can’t access your website because they will always redirect to https site.

Now, you can go to http://www.ssllabs.com to test your website. You should be able to get score A+