Varnish VMOD static code analysis

I recently went looking for something similar to pep8/pylint when writing Varnish VMODs, and ended up with OCLint.

I can’t really speak to how good it is, but it catches the basic stuff I was interested in.

The documentation is mostly for cmake, so I’ll give a small tutorial for automake:

  • (download+install oclint to somewhere in $PATH)
  • apt-get install bear
  • cd libvmod-xxx
  • ./autogen.sh; ./configure –prefix=/usr
  • bear make # “build ear” == bear. writes compile_commands.json
  • cd src
  • oclint libvmod-xxx.c # profit
  • Which will tell you about unused variables, useless parentheses, dead code and so on.

    Posted in varnish | Tagged , , , | Leave a comment

    Announcing libvmod-tcp: Adjust Varnish congestion control algorithm.

    I’ve uploaded my new TCP VMOD for Varnish 4 to github, you can find it here:
    http://github.com/lkarsten/libvmod-tcp.

    This VMOD allows you to get the estimated client socket round trip time, and then let you change the TCP connection’s congestion control algorithm if you’re so inclined.

    Research[tm][0] says that Hybla is better for long high latency links, so currently that is what it is used for.

    Here is a quick VCL example:

    if (tcp.get_estimated_rtt() > 300) {
    set req.http.x-tcp = tcp.congestion_algorithm("hybla");
    }

    One thing to note is that VCL handling is very early in the TCP connection lifetime. We’ve only just read and acked the HTTP request. The readings may be off, I’m analyzing this currently.
    (As I understand it the Linux kernel will keep per-ip statistics, so for subsequent requests this should get better and better..)

    References:
    0: Esterhuizen, A., and A. E. Krzesinski. “TCP Congestion Control Comparison.” (2012).

    Posted in varnish | Tagged , , , | Leave a comment

    Fresh Varnish packages for Debian/Ubuntu and Redhat systems

    We use continuous integration when developing Varnish Cache. This means that we run our internal test suite (varnishtest) on all commits, so we catch our mistakes earlier.

    This pipeline of build jobs sometimes end up with binary packages of Varnish, which may be useful to people when they know they exist. They may not be the easiest to find, which this blog post tries to remedy.

    Development wise, Varnish Cache is developed with GIT with a master branch for development and a set of production branches, currently 3.0 and 4.0.

    Unreleased packages for Varnish master can be found here: https://jenkins.varnish-software.com/view/varnish-master/

    Unreleased packages of Varnish 4.0 can be found here: https://jenkins.varnish-software.com/view/varnish-4.0/

    (There is also a set of 3.0 jobs, but you should really go for 4.0 these days.)

    The latest commits in each of the production branches may contain fixes we’ve added after the last production release, but haven’t cut a formal release for yet. (For example there are some gzip fixes in the 3.0 branch awaiting a 3.0.6 release, which I really should get out soon.)

    Some jobs in the job listing just check that Varnish builds, without creating any output (or artifacts as Jenkins calls it.) This applies for any jobs with “-build-” in the name, for example varnish-4.0-build-el7-x86_64 and varnish-4.0-build-freebsd10-amd64.

    The Debian and Ubuntu packages are all built from one job currently, called varnish-VERSION-deb-debian-wheezy-amd64. Press “Expand all” under artifacts to get the full list.

    Redhat/RHEL packages are built in the different el5/el6/el7 jobs.

    The unreleased packages built for 3.0 and 4.0 are safe. This is the process used to build the officially released packages, just a step earlier in the process. The varnish-master packages are of course failing from time to time, but that is to be expected.

    The version numbers in the packages produced may be a bit strange, but that is what you get with unreleased software builds.

    I’m happy to improve this process and system if it can help you run never versions of Varnish, comments (either here or on IRC) are appreciated.

    Posted in stuff, varnish | Tagged , , | 6 Comments

    Disable Spotify song change notification in Debian Linux

    Recently Spotify started sending notifications to the desktop on song change. This is unnecessarily annoying and breaks my flow, so it had to go.

    (and since I usually just listen to the same playlists anyway, I’m very well aware what the song name is ;-))

    This appeared after a recent apt-get upgrade on my Debian Jessie machine. The interwebs is full of helpful advice on how to fix this on your Mac (growl something something), but not so much for Debian Linux.

    The nice Archlinux people knew how, however:

    After version 0.9.10, track change notifications were enabled by default. They can be quite intrusive. To disable them, add the following line to ~/.config/spotify/Users/<spotifylogin>-user/prefs

    ui.track_notifications_enabled=false
    It is also possible to launch spotify with the –ui.track_notifications_enabled=false option.

    Works like a charm. Perfect. I’m using version 0.9.10.17.g4129 of spotify-client from their apt-repository.

    Posted in Uncategorized | Tagged , , , | 1 Comment

    What happened to ban.url in Varnish 4.0?

    tl;dr; when using Varnish 4 and bans via varnishadm, instead of “ban.url EXPRESSION”, use “ban req.url ~ EXPRESSION”.

    In Varnish 3.0 we had the ban.url command in the varnishadm CLI. This was a shortcut function expanding to the a bit cryptic (but powerful) ban command. In essence ban.url just took your expression, prefixed it with “req.url ~ ” and fed it to ban. No magic.

    We deprecated this in Varnish 4.0, and now everyone has to update their CMS’s plugin for cache  invalidation. Hence this blog post. Perhaps it will help. Perhaps not. :-)

    Some references:

    Posted in stuff | Tagged , , | Leave a comment

    Converting a Varnish 3.0 VMOD to 4.0

    So we’re getting closer to releasing the first proper 4.0 version of Varnish Cache. One of the things we need to fix is to get all the vmod writers to make sure their vmod works with the new version.

    Here are my notes from doing just that, in the hope to make it simpler for others.

    In 4.0, you don’t need the source tree of Varnish any more. The include files will be enough, and pkg-config will find them for you.

    Make sure that /usr/lib/pkgconfig/varnishapi.pc and /usr/share/aclocal/varnish.m4 exists. If you installed Varnish in the standard path/prefix, that should work out of the box. Otherwise, you might to add some symlinks for pkg-config and automake to find the source. (I need multiple varnishd versions when debugging customer problems, so I let them live in /opt/varnishX.Y/ on my laptop)

    Pull/merge the new Makefile.am files from the master branch of libvmod-example.

    Header files: remove bin/varnishd/cache.h and add cache/cache.h.

    Vmod functions are now called with a vrt context as first argument. %s/struct sess \*sp/const struct vrt_ctx \*ctx/g

    The old sess struct has been split, some data is in vrt_ctx->req, and some is in vrt_vtx->req->sp. Look up what is where in cache/cache.h. 

    I’ve put up the 3.0->4.0 diff for vmod_policy.c as a gist: https://gist.github.com/lkarsten/8039861

    There was a bit trouble of finding varnishtest, as src/Makefile was missing the reference entirely. I just fixed it by hand for now. Another thing for the 4.0 todolist, then.

    And finally; 

    lkarsten@immer:~/work/libvmod-policy/src$ make check
    /opt/varnish/bin/varnishtest -Dvarnishd=/opt/varnish/sbin/varnishd -Dvmod_topbuild=/home/lkarsten/work/libvmod-policy tests/test01.vtc
    # top TEST tests/test01.vtc passed (1.574)

     

    I have a working Varnish 4.0 vmod. :-D

    Posted in Uncategorized | Tagged , , | 2 Comments

    DNS RBL test address for development

    If you are writing code that checks a DNS real-time blockhole list (RBL), it looks like 127.0.0.2 is the standard address that is always in the black/white -list.

    This is probably know for most sysadmins/security people and whatnot, but wasn’t entirely trivial to find using Google.

    lkarsten@immer:~$ dig 2.0.0.127.dnsbl.sorbs.net @8.8.8.8
    ; <<>> DiG 9.8.4-rpz2+rl005.12-P1 <<>> 2.0.0.127.dnsbl.sorbs.net @8.8.8.8
    ;; global options: +cmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55083
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 0
    ;; QUESTION SECTION:
    ;2.0.0.127.dnsbl.sorbs.net. IN A
    ;; ANSWER SECTION:
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.10
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.5
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.7
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.2
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.3
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.9
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.14
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.4
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.6
    2.0.0.127.dnsbl.sorbs.net. 2562 IN A 127.0.0.8
    ;; Query time: 17 msec
    ;; SERVER: 8.8.8.8#53(8.8.8.8)
    ;; WHEN: Wed Dec 11 14:12:20 2013
    ;; MSG SIZE rcvd: 203
    lkarsten@immer:~$

    Good to be able to actually test your code for hits also.

    (this is for libvmod-policy, so you can deny/reject POST/PUT from spammers in Varnish)

    Posted in stuff | Tagged , , , , | Leave a comment

    Varnish and Ghost blogging software

    So there is a new shiny blogging platform out called Ghost. Looks pretty good to me.

    If you want to run it behind Varnish, you’ll soon notice it has the usual problem of setting session cookies everywhere leading to 0% hit rate. 

    I have written a Varnish VCL configuration for filtering this in the necessary places, while keeping the admin interface working still.

    You can find it here:

    https://gist.github.com/lkarsten/6683179

    Have fun.

    Posted in stuff | Tagged , , | 1 Comment

    Testing VMODs with Travis (travis-ci.org)

    Travis CI is a service where open source software can run tests automatically on commits. It hooks into github in a silky smooth way.

    If you’re developing a Varnish module (VMOD), you probably have started out with our libvmod-example package. It has all the automake magic you need, as well as some simple test cases to test your vmod. Given that you’ve written som varnishtest testcases for it (you really should), you now can get travis to run them as well!

    I’ve put a small recipe for this into the libvmod-example package.

    Feel free to play around with it, feedback appreciated. For the travis setup bits, consult the travis getting started guide. The final result is something like this, shown for libvmod-cookie:

    https://travis-ci.org/lkarsten/libvmod-cookie

    Posted in stuff | Tagged , , , , , | Leave a comment

    GSM A5/1 rainbow tables in Oslo, Norway

    The A5/1 encryption algorithm used in (traditional) GSM networks were proven to be breakable by brute force back in 2009/2010. This means that GSM calls can be intercepted, decoded and listened to by anyone. (SMS also, but that is a different story)

    To do this easily you need a 1600 GB big rainbow table.

    These files are/were available on bittorrent; a mirror of the torrents is available. The original source on reflextor.com is unavailable.

    I’m offering the files for sale on disk or as paid download.

    The intent with this is to facilitate further research and experimentation. Drop me an email and we’ll figure out the details.

    (PS: I still have the Ettus URSP2 with all daughterboards/cables/antennas needed to play around with this. I don’t use much any longer, available for sale/donation to a good cause..)

    Updated 2013-10-23: Clarified language.

    Posted in stuff | Tagged , , , , , , | 35 Comments