PROXY protocol in Varnish

Dag has been working implementing support for HAProxy’s PROXY protocol[1] in Varnish. This is a protocol adds a small header on each incoming TCP connection that describes who the real client is, added by (for example) an SSL terminating process. (since srcip is the terminating proxy)

We’re aiming for merging this into Varnish master (so perhaps in 4.1?) when it is ready.

The code is still somewhat unfinished, timeouts are lacking and some polishing needed, but it works and can be played with in a development setup.

Code can be found here: https://github.com/daghf/varnish-cache/tree/PROXY

I think Dag is using haproxy to test it with. I’ve run it with stunnel (some connection:close issues to figure out still), and I’d love if someone could test it with ELB, stud or other PROXY implementations.

1: http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt

Advertisement

Announcing libvmod-tcp: Adjust Varnish congestion control algorithm.

I’ve uploaded my new TCP VMOD for Varnish 4 to github, you can find it here:
http://github.com/lkarsten/libvmod-tcp.

This VMOD allows you to get the estimated client socket round trip time, and then let you change the TCP connection’s congestion control algorithm if you’re so inclined.

Research[tm][0] says that Hybla is better for long high latency links, so currently that is what it is used for.

Here is a quick VCL example:

if (tcp.get_estimated_rtt() > 300) {
set req.http.x-tcp = tcp.congestion_algorithm("hybla");
}


One thing to note is that VCL handling is very early in the TCP connection lifetime. We’ve only just read and acked the HTTP request. The readings may be off, I’m analyzing this currently.
(As I understand it the Linux kernel will keep per-ip statistics, so for subsequent requests this should get better and better..)

References:
0: Esterhuizen, A., and A. E. Krzesinski. “TCP Congestion Control Comparison.” (2012).

Disable Spotify song change notification in Debian Linux

Recently Spotify started sending notifications to the desktop on song change. This is unnecessarily annoying and breaks my flow, so it had to go.

(and since I usually just listen to the same playlists anyway, I’m very well aware what the song name is ;-))

This appeared after a recent apt-get upgrade on my Debian Jessie machine. The interwebs is full of helpful advice on how to fix this on your Mac (growl something something), but not so much for Debian Linux.

The nice Archlinux people knew how, however:

After version 0.9.10, track change notifications were enabled by default. They can be quite intrusive. To disable them, add the following line to ~/.config/spotify/Users/<spotifylogin>-user/prefs

ui.track_notifications_enabled=false
It is also possible to launch spotify with the –ui.track_notifications_enabled=false option.

Works like a charm. Perfect. I’m using version 0.9.10.17.g4129 of spotify-client from their apt-repository.

What happened to ban.url in Varnish 4.0?

tl;dr; when using Varnish 4 and bans via varnishadm, instead of “ban.url EXPRESSION”, use “ban req.url ~ EXPRESSION”.

In Varnish 3.0 we had the ban.url command in the varnishadm CLI. This was a shortcut function expanding to the a bit cryptic (but powerful) ban command. In essence ban.url just took your expression, prefixed it with “req.url ~ ” and fed it to ban. No magic.

We deprecated this in Varnish 4.0, and now everyone has to update their CMS’s plugin for cache  invalidation. Hence this blog post. Perhaps it will help. Perhaps not. :-)

Some references:

• https://www.varnish-cache.org/docs/trunk/users-guide/purging.html#bans – the proper docs on ban. Note the part about lurker friendliness.
• https://www.varnish-cache.org/lists/pipermail/varnish-dev/2012-November/007399.html  – the discussion on the mailing list on removing it.

Converting a Varnish 3.0 VMOD to 4.0

So we’re getting closer to releasing the first proper 4.0 version of Varnish Cache. One of the things we need to fix is to get all the vmod writers to make sure their vmod works with the new version.

Here are my notes from doing just that, in the hope to make it simpler for others.

In 4.0, you don’t need the source tree of Varnish any more. The include files will be enough, and pkg-config will find them for you.

Make sure that /usr/lib/pkgconfig/varnishapi.pc and /usr/share/aclocal/varnish.m4 exists. If you installed Varnish in the standard path/prefix, that should work out of the box. Otherwise, you might to add some symlinks for pkg-config and automake to find the source. (I need multiple varnishd versions when debugging customer problems, so I let them live in /opt/varnishX.Y/ on my laptop)

Pull/merge the new Makefile.am files from the master branch of libvmod-example.

Header files: remove bin/varnishd/cache.h and add cache/cache.h.

Vmod functions are now called with a vrt context as first argument. %s/struct sess \*sp/const struct vrt_ctx \*ctx/g

The old sess struct has been split, some data is in vrt_ctx->req, and some is in vrt_vtx->req->sp. Look up what is where in cache/cache.h. 

I’ve put up the 3.0->4.0 diff for vmod_policy.c as a gist: https://gist.github.com/lkarsten/8039861

There was a bit trouble of finding varnishtest, as src/Makefile was missing the reference entirely. I just fixed it by hand for now. Another thing for the 4.0 todolist, then.

And finally; 

lkarsten@immer:~/work/libvmod-policy/src$ make check
/opt/varnish/bin/varnishtest -Dvarnishd=/opt/varnish/sbin/varnishd -Dvmod_topbuild=/home/lkarsten/work/libvmod-policy tests/test01.vtc
# top TEST tests/test01.vtc passed (1.574)

 

I have a working Varnish 4.0 vmod. :-D

DNS RBL test address for development

If you are writing code that checks a DNS real-time blockhole list (RBL), it looks like 127.0.0.2 is the standard address that is always in the black/white -list.

This is probably know for most sysadmins/security people and whatnot, but wasn’t entirely trivial to find using Google.

lkarsten@immer:~$ dig 2.0.0.127.dnsbl.sorbs.net @8.8.8.8
; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> 2.0.0.127.dnsbl.sorbs.net @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55083
;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;2.0.0.127.dnsbl.sorbs.net. IN A
;; ANSWER SECTION:
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.10
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.5
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.7
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.2
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.3
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.9
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.14
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.4
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.6
2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.8
;; Query time: 17 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Dec 11 14:12:20 2013
;; MSG SIZE rcvd: 203
lkarsten@immer:~$

Good to be able to actually test your code for hits also.

(this is for libvmod-policy, so you can deny/reject POST/PUT from spammers in Varnish)

Varnish and Ghost blogging software

So there is a new shiny blogging platform out called Ghost. Looks pretty good to me.

If you want to run it behind Varnish, you’ll soon notice it has the usual problem of setting session cookies everywhere leading to 0% hit rate. 

I have written a Varnish VCL configuration for filtering this in the necessary places, while keeping the admin interface working still.

You can find it here:

https://gist.github.com/lkarsten/6683179

Have fun.

GSM A5/1 rainbow tables in Oslo, Norway

The A5/1 encryption algorithm used in (traditional) GSM networks were proven to be breakable by brute force back in 2009/2010. This means that GSM calls can be intercepted, decoded and listened to by anyone. (SMS also, but that is a different story)

To do this easily you need a 1600 GB big rainbow table.

These files are/were available on bittorrent; a mirror of the torrents is available. The original source on reflextor.com is unavailable.

I’m offering the files for sale on disk or as paid download.

The intent with this is to facilitate further research and experimentation. Drop me an email and we’ll figure out the details.

Updated 2013-10-23: Clarified language.

Update 2016-05-10: Too much hassle shipping disk drives from outside the EU. From this point on I’m offering access to the files on FTP only.

Building a Varnish VMOD on Debian

From the tutorials department, here are some quick notes on how to install a Varnish VMOD from source.

This is slightly complicated because Varnish demands that a VMOD must be built against the same git commit (or release) as the one that is running. This will be relaxed in future versions.

Current setup is a standalone Varnish VM on Debian Wheezy with Varnish installed from varnish package archives (3.0.4-1~wheezy.)

1. Get the vmod

lkarsten@lb1:~$ git clone https://github.com/lkarsten/libvmod-cookie.git
Cloning into 'libvmod-cookie'...
remote: Counting objects: 253, done.
remote: Compressing objects: 100% (131/131), done.
remote: Total 253 (delta 132), reused 232 (delta 112)
Receiving objects: 100% (253/253), 49.51 KiB, done.
Resolving deltas: 100% (132/132), done.
lkarsten@lb1:~$

2. Get and configure the source tree for the running Varnish

Verify first that you have the necessary package repositories enabled:

lkarsten@lb1:~$ grep varnish /etc/apt/sources.list
deb http://repo.varnish-cache.org/debian/ wheezy varnish-3.0
deb-src http://repo.varnish-cache.org/debian/ wheezy varnish-3.0
lkarsten@lb1:~$

After that, continue with the juicy parts:

lkarsten@lb1:~$ apt-get source varnish 
Reading package lists... Done 
Building dependency tree 
Reading state information... Done 
NOTICE: 'varnish' packaging is maintained in the 'Git' version control system at: 
git://git.debian.org/pkg-varnish/pkg-varnish.git 
Need to get 2,060 kB of source archives. 
Get:1 http://repo.varnish-cache.org/debian/ wheezy/varnish-3.0 varnish 3.0.4-1 (dsc) [2,334 B] 
Get:2 http://repo.varnish-cache.org/debian/ wheezy/varnish-3.0 varnish 3.0.4-1 (tar) [2,044 kB] 
Get:3 http://repo.varnish-cache.org/debian/ wheezy/varnish-3.0 varnish 3.0.4-1 (diff) [14.1 kB] 
Fetched 2,060 kB in 0s (11.4 MB/s) 
gpgv: keyblock resource `/home/lkarsten/.gnupg/trustedkeys.gpg': file open error 
gpgv: Signature made Fri 14 Jun 2013 11:56:48 CEST using RSA key ID 87218D9C 
gpgv: Can't check signature: public key not found 
dpkg-source: warning: failed to verify signature on ./varnish_3.0.4-1.dsc 
dpkg-source: info: extracting varnish in varnish-3.0.4 
dpkg-source: info: unpacking varnish_3.0.4.orig.tar.gz 
dpkg-source: info: applying varnish_3.0.4-1.diff.gz 
lkarsten@lb1:~$
lkarsten@lb1:~$ cd varnish-3.0.4
lkarsten@lb1:~/varnish-3.0.4$ ./autogen.sh
[..]
lkarsten@lb1:~/varnish-3.0.4$ ./configure --prefix=/usr
[..]
lkarsten@lb1:~/varnish-3.0.4$ make

If configure or make fails, you might need some additional packages. Run an apt-get build-dep varnish and work from there. (if editline fails on you, remember to rerun configure after installing it)

3. Build and install the vmod

lkarsten@lb1:~$ cd libvmod-cookie/
lkarsten@lb1:~/libvmod-cookie$ ./autogen.sh
+ aclocal -I m4
+ libtoolize --copy --force
libtoolize: putting auxiliary files in `.'.
libtoolize: copying file `./ltmain.sh'
libtoolize: putting macros in AC_CONFIG_MACRO_DIR, `m4'.
libtoolize: copying file `m4/libtool.m4'
libtoolize: copying file `m4/ltoptions.m4'
libtoolize: copying file `m4/ltsugar.m4'
libtoolize: copying file `m4/ltversion.m4'
libtoolize: copying file `m4/lt~obsolete.m4'
+ autoheader
+ automake --add-missing --copy --foreign
configure.ac:8: installing `./config.guess'
configure.ac:8: installing `./config.sub'
configure.ac:11: installing `./install-sh'
configure.ac:11: installing `./missing'
src/Makefile.am: installing `./depcomp'
+ autoconf
lkarsten@lb1:~/libvmod-cookie$ 
lkarsten@lb1:~/libvmod-cookie$ ./configure VARNISHSRC=~/varnish-3.0.4/
[..]
# and finally
lkarsten@lb1:~/libvmod-cookie$ make
[..]
libtool: link: ( cd ".libs" && rm -f "libvmod_cookie.la" && ln -s "../libvmod_cookie.la" "libvmod_cookie.la" )
make[2]: Leaving directory `/home/lkarsten/libvmod-cookie/src'
make[2]: Entering directory `/home/lkarsten/libvmod-cookie'
rst2man README.rst vmod_cookie.3
make[2]: Leaving directory `/home/lkarsten/libvmod-cookie'
make[1]: Leaving directory `/home/lkarsten/libvmod-cookie'
lkarsten@lb1:~/libvmod-cookie$ 
lkarsten@lb1:~/libvmod-cookie$ sudo make install
[..]
/bin/mkdir -p '/usr/local/share/man/man3'
 /usr/bin/install -c -m 644 vmod_cookie.3 '/usr/local/share/man/man3'
make[2]: Leaving directory `/home/lkarsten/libvmod-cookie'
make[1]: Leaving directory `/home/lkarsten/libvmod-cookie'
lkarsten@lb1:~/libvmod-cookie$

At this point you should have the two vmod files available for Varnish:

lkarsten@lb1:~/libvmod-cookie$ ls -l /usr/lib/varnish/vmods/
total 64
-rwxr-xr-x 1 root root 966 Jul 29 11:11 libvmod_cookie.la
-rwxr-xr-x 1 root root 41538 Jul 29 11:11 libvmod_cookie.so
-rw-r--r-- 1 root root 16128 Jun 17 13:38 libvmod_std.so
lkarsten@lb1:~/libvmod-cookie$

And you are done!

“import cookie” should now work without issue in your /etc/varnish/default.vcl.

Setting client.ip in Varnish VCL with libvmod-ipcast

I’ve written a new Varnish 3.0 VMOD called ipcast.

It has a single function; ipcast.clientip(ipstring) which sets the internal Varnish variable client.ip to whatever IPv4/IPv6 address you give as the argument.

You need this if you want to do ACL checks on connections done through a load balancer or SSL terminator. In those cases client.ip would be 127.0.0.1 and you get the real client’s IP address in the X-Forwarded-For (or similar) header.

You can find it here:

https://github.com/lkarsten/libvmod-ipcast

Here is some example VCL to illustrate how it works. I think the regex needs some work, suggestions/pull requests are welcome.

import ipcast;
acl friendly_network {
    "192.0.2.0"/24;
}
sub vcl_recv {
    if (req.http.X-Forwarded-For !~ ",") {
        set req.http.xff = req.http.X-Forwarded-For;
    } else {
        set req.http.xff = regsub(req.http.X-Forwarded-For,
                "^[^,]+.?.?(.*)$", "\1");
    }

    if (ipcast.clientip(req.http.xff) != 0) {
        error 400 "Bad request";
    }

    if (client.ip !~ friendly_network) {
            error 403 "Forbidden";
    }
}