Internet routing with Quagga/Linux on Free Internet eXchange Oslo (FIXO)

I run a small internet routing AS (autonomous system), mostly to keep my routing skills up to date (somewhat).

AS56809 get transit from Blix Solutions AS, and it is also present on Free Internet eXchange Oslo (FIXO for short). FIXO can be compared to the somewhat bigger NIX peering point also in Oslo.

If you want to run BGP with Quagga on FIXO, here are a few hints on how to do it. This is made with quagga 0.99.14 from debs on debian wheezy. Let me add that I’m fairly inexperienced at this stuff, so there might be bad advise in this post :)

Make some access lists your networks in it, handy for filtering what routes you will be sending to peers:

conf t
access-list as64496-networks permit 192.0.2.0/24
access-list as64496-networks deny deny

ipv6 prefix-list ipv6-as64496-networks seq 2 permit 2001:db8::/32
ipv6 prefix-list ipv6-as64496-networks seq 10 deny any

Basic quagga/BGP setup:

conf t
password <pw>
ip forwarding
ipv6 forwarding

router bgp 64496
bgp router-id x.x.x.x
network 192.0.2.0/24
address-family ipv6
network 2001:db8::/32

Make some peer groups that contain most of the boilerplate stuff:

router bgp 64496
neigh fixo-peers maximum-prefix 100
neigh fixo-peers soft-reconfiguration inbound
neigh fixo-peers distribute-list as64496-networks out

address-family ipv6
neighbor fixo-peers activate
neighbor fixo-peers soft-reconfiguration inbound
neighbor fixo-peers maximum-prefix 100
neighbor fixo-peers prefix-list ipv6-as64496-networks out

Adding a new IPv4 peering:

conf t
router bgp 64496
neigh 91.198.176.x remote-as yyyy
neigh 91.198.176.x peer-group fixo-peers
neigh 91.198.176.x desc foo@bar.com, +47 1234

Adding a new IPv6 peering:

conf t
router bgp 64496
neigh 2001:7f8:41:0:xxxx:1 remote-as yyyy
neigh 2001:7f8:41:0:xxxx:1 desc foo@bar.com, +47 1234

address-family ipv4 unicast
no neigh 2001:7f8:41:0:xxxx:1 activate
address-family ipv6
neigh 2001:7f8:41:0:xxxx:1 peer-group fixo-peers

Making sure it works

After you’ve set things up, you need to verify that the peering comes up (Established state), and that you’re sending (and receiving) the routes you intend. A common mistake is to send the peer your whole routing table, which is why the distribute-list/prefix-list is in there.

Some handy commands are:

# show how this peering is doing. you want Established.
show ip bgp neigh 91.198.176.xx
# show what routes you are sending to this peer, should be the 1-5 routes you have in the as64496-networks access list.
show ip bgp nei 91.198.176.xx advertised-routes
# what you are getting from the peer. Tab-completion in vtysh is a pain on this command.
show ip bgp neigh 91.198.176.xx received-routes

# reset the peering.
clear ip bgp neigh 91.198.176.xx

If you are running IPv6 peerings, you’ll soon notice that the CLI commands are quite incoherent.

# general overview is under the ordinary ip bgp listing:
show ip bgp nei 2001:7f8:41:0:xxxx:1

# but, if you want to see advertised (and received) routes, these are here:
show ipv6 bgp nei 2001:7f8:41:0:xxxx:1 advertised-routes

Route server

Fixo operates a set of route servers. This is in essence BGP routers that only contains routes, and passes leaves the next-server in announcements as they were. In essence it means that you peer with the route server and get all the routes of all the others that also peer with it. Saves you a lot of emails.

Setting up the peering is identical to any ipv4/ipv6 peering, except that you need to tag/mark the routes you want redistributed by it with a community.

route-map fixo-routeserver-out permit 10 
 set community 61300:61300

Quagga doesn’t allow you to set a route-map on a single member of a peer-group, so you will have to duplicate some config for these two peers.

neighbor 91.198.176.253 remote-as 61300
neighbor 91.198.176.253 description fixo-routeserver-1
neighbor 91.198.176.253 soft-reconfiguration inbound
neighbor 91.198.176.253 maximum-prefix 300
neighbor 91.198.176.253 distribute-list as64496-networks out
neighbor 91.198.176.253 route-map fixo-routeserver-out out

All done!

Guidelines for ticketing systems (rant)

If you are writing a ticketing/support request system, make sure to remember the following:

• It is ok to mangle the subject line if you are the ticketing system.
• Absolutely no UTF8 support. ø Ã¥ ought to do it.
• Make sure to include all of the ticket information when notifying the requestor of ticket state change.
• Do not indicate which part has actually changed, the requestor can guess this easily.
• All time references should be in local time without any indication of time zone. The requestor will know where you do business and where the support staff is working from.
• Only insane users has wider screens than 800 pixels. (Hello, osTicket.)

If you are writing the interface used by the support personnel:

• Any web page must have at least 60 links with similar names.
• Make a bad re-implementation of SQL as query language. Require this for any searches for old tickets. (Yes, I am looking at you Request Tracker.)
• Text fields for writing customer replies in must not disable Ctrl-W or ESC, so that bash/vim users are rightfully punished when using muscle memory key bindings.
• Page loads should take at the minimum 10 seconds.
• Always show full ticket history, especially that the system has sent copies of email to all involved. The most recent information should be the last to load.
• Use javascript to render the menu bar after everything else is done. This keeps the user from skipping to another ticket too soon.

It is the law that all ticketing systems must adhere to this golden standard.

Varnish paywall presentation

I held a technical presentation for Redpill Linpro about running a paywall/metered access with Varnish yesterday.

Slides as PDF are here: RL-paywall-2013-03

Main talking points were how it runs within Varnish, what extra vmods are in use, and finally how it is integrated with your backend/cms.

To keep the VS marketing department happy (Hi!), here is the link to our product page: https://www.varnish-software.com/varnish-paywall :-)

 

 

Varnish development news (Feb 2013)

An update from git:

• Tab completion in varnishadm. (e6a88da)
• TCP_KEEPALIVE is now enabled for all connections on operating systems that supports it. This avoids a situation where Varnish would keep retrying failing slow backend request for clients that had given up and closed the connection. (f91fc4d)
• [3.0] The TCP_NODELAY patch introduced in September was backported into 3.0. This disables Nagle’s algorithm on connections, and avoids a situation where the last chunk of a chunked response from ESI/internal gzip got buffered by the server’s IP stack before being sent to the client. (ef5e9e0)
• Default VCL now takes Surrogate-Control: no-store and Cache-Control: no-cache/no-store/private into account when computing TTL on a backend response. (81006ea)
• Per took a stab at the documentation, rework underway to fit Scoof’s proposed new model.

We (Varnish Software) are looking for a junior frontend developer in Oslo, Norway. Skiing skills optional. The web page might not be updated yet, but please apply here: https://www.varnish-software.com/jobs

HTTP Live Streaming (HLS) tools

Delivering video over HTTP is up and coming. The two protocols that seem to be winning (currently) are HLS and HDS. These are my notes from playing around with HLS.

If you want to play HLS on Linux, this python player works: http://gitorious.org/hls-player . Sometimes recent VCL 2.0.3 works, but not always (and the progress bar behaves very strangely..)

If you want to dump an HLS stream to a single .ts file, use this tool: https://github.com/osklil/hls-fetch

If you want to dump a HLS VOD (video on demand, premade) stream to local disk to play with; try out my quick hls-mirror script: https://gist.github.com/anonymous/4755742

If you want to segment some media file for later video-on-demand delivery, using bleeding edge avconv (with just libx264+lamemp3) from git:

avconv -i big_buck_bunny_720p_surround.avi -vcodec h264 -acodec mp3 -hls_time 10 -hls_list_size 999999999 foo/output.m3u8

hls_list_size seems to be how many items (of the n made) that should appear in the m3u8 file. Setting to 0 actually means an empty m3u8, so useful!

The outputted files works on my android 4.1 and on my desktop.

Why all this you ask? Because video over HTTP works very well through Varnish, even better with streaming (cut-through forwarding) enabled. Even on mediocre DSL you can make a continous stream/file set, push to a Varnish on AWS/whatever, and anyone can broadcast live to the world. Slightly delayed, but standards based, simple to do and simple to scale.

I think we’ve just seen the start of how cool (and disrupting) this will become.

Varnish Cache development news

Short update on activities in Varnish cache development:

• Work is being done on getting access to the request BODY inside Varnish. This feature has been requested for a while, particularly by the security.vcl guys. See daf0e2c.
• The 2013Q1 Varnish Developer Day will be held in København February 4th.
• I had my request approved and got req.request (which was really the HTTP method) renamed to req.method. Hurray. 0e9627d
• Martin is steadily working on a new libvarnishapi, that makes writing varnish tools (top, stat, ncsa..) simpler. Better filtering language, more stable and better performance. Or something like that, the details are still fuzzy. :)
• I’ve been running varnish trunk in “production” on hyse.org for a couple of weeks now. No ill effects in serving data to report. Works fine for my (minor) needs. One small note is that varnishlog fd/request grouping doesn’t work currently so you need “varnishlog -O” to get any useful output at all. The rest of the tools are probably broken as well. :)
• On the stable side of Varnish, we (Varnish Software) have got some experience with 3.0.3 now. It seems pretty solid. Setting a backend administratively sick from varnishadm is useful.
• The Varnish Agent 2.0 was released the other day. It is a RESTful interface in front of varnishadm, and lets you query/set stuff over HTTP. We will use it for our management tool (VAC), but it should be useful for other uses as well. Source and release announcement.

Anonymous contributions with git

I’ve been playing around with the idea of how to make anonymous contributions to open source software.

Let us say that you want to contribute to a project, but it is not something you necessarily want to be associated with. The git tree will (probably) last for the next 10+ years, probably enough that it could influence your career options.

The project could be the tor project, or maybe metasploit? Really anything above average controversial, but you might want that job at $risk_averse_bank later on.

First of you would start out with setting your username and email in your cloned copy. git config user.email anonymous@example.com, and git config user.name Anonymous. Cool, future commits are not made as the default user/email, but overridden inside this repository.

But how do you get that golden patch set you’ve written upstream?

git format-patch + email to the development list (if it exists) is a possibility. I haven’t looked into sending anonymous emails for quite a while, but I suspect it isn’t trivial to get it right.

You can pastebin the patches somewhere, and somehow ask the developers to merge it.

All in all, this is probably things that can be solved. A bit of research, maybe a bit of tor, and it is all good. Or at least gives sufficient anonymity for fixing that typo in a readme file somewhere.

But it is cumbersome. You just want to fix that typo. The sweetness of the github fork+fix+pullrequest is that it is very low effort for the contributer.

So I’ve bounced the idea around to friends and coworkers, to see if someone feels this need and how it can be solved without burdening the commiter to badly. There are two ideas on the table;

1. Implement some sort of extra transport inside git that takes a repository name, hashes it, finds some public storage (think DHT) out there and pushes in that direction. Ignoring the bad connotations, this might just be freenet all over again. The storage network can be community driven, tends to the DHT and syncs git trees back and forth for robustness/reliability. If the user needs extra anonymity, use tor to do the pushes. (this needs more work)

2. Make an open github-style push target that just accepts any repository. Push to git://anongit.com/$RANDOM/. If the tree isn’t there, the service runs git-init and accepts. Go to http://anongit.com/$RANDOM/ and get a github/gitweb(uh) style web page for your (secret) repo. The web page also has a list of commiters (from the git log), and users of the web interface can ask the service to send a pull request over email for you.

Implementing #2 shouldn’t be too hard, but running one of these servers over time might be. (abuse, takedown notices..) Doing #1, perhaps with .onion services inside tor might work, but requiring tor for access makes things more complicated for the user.

Are there more options you can think of?

Things that might influence that I haven’t touched into in this post; 1) do upstream devs really want to accept patches from clearly anonymous sources? 2) should it be anonymous <anonymous@example.com>, or something less conspicuous? 3) should the commit time stamp be changed to 1970? 4) writing style/text analysis of the commit message might give you away. 5) maybe people only send pull requests for the fame. In that case anonymous commits doesn’t apply well.

Varnish trick: Serve stale content while refetching

Here is a small trick we recently implemented for a customer:

The main premise was:

No clients should have to wait while the backend works. If a request is a miss, give the client a slightly stale/old page instead, and fetch a new page in the background.

Since Varnish and VCL is super configurable, we can do this with a VCL hack and a small helper process.

The flow is that a client requests something that just expired. In vcl_miss we notice this, and change the backend to a sick one. We also log the URL that just failed with std.log(), before restarting the request handling. Back in vcl_recv the usual sick-backend behaviour kicks in and the slightly stale graced object is given to the client.

Outside Varnish there is a small python script that tails varnishlog output for a special VCL_Log entry. When it picks it up, it sends a request for the same URL to the local Varnish. In vcl_recv we detect this client, set req.hash_always_miss to force a refetch and let the python script wait while the backend recreates the page.

All requests that come in while the refetch is underway will be served graced copies at full speed.

Your 95 percentile response time graphs will love this feature, and maybe even some of your users as well. Cool, huh?

More information about grace in Varnish:

https://www.varnish-software.com/static/book/Saving_a_request.html#core-grace-mechanisms

New Varnish VMOD: Softer purges / invalidations

With the new softpurge vmod you can do cache invalidation in Varnish that only affects objects if your backend is up. This means that you can purge all you want, and in the normal case everything works as expected, but if your backend/origin server is unhappy you can serve stale content instead of a 503.

It introduces softpurge.softpurge(); where you would normally use purge; (in vcl_hit and vcl_miss).

The trick is of course that we only reduce the TTL of cached objects (and their Vary-ants), but keep the grace period. This gives us the additional win of increased concurrency while your backend recreates the content after a purge, all but the first client requesting a page withing the page creation time is served the stale one, per usual varnish grace behaviour.

You can find the vmod on my github account. Please remember that this is an early version which has not seen heavy production use yet.

PS: We’re looking into doing softer bans (the other way of doing cache invalidation in Varnish) as well. This looks to be a bit more involved, so we (Varnish Software) are looking for sponsors for this task. Ping me (or Ruben) if this sounds interesting.

Varnishncsa and std.log()

New in Varnish 3.0.3rc1 is that you can put arbitrary log lines from VCL into the varnishncsa output. This can be used for funky stuff like logging the session cookie along with the request.

Let’s say you have the following VCL and have a session id as the single site cookie:

import std;

sub vcl_deliver {

std.log(“sessionid:” + regsub(req.http.cookie, “.*=”, “”);

}

Run varnishncsa as follows:

 # varnishncsa -F ‘%h %l %u %t “%r” %s %b “%{Referer}i” “%{User-agent}i %{VCL_Log:sessionid}x’

And now your access logs contains the session id/cookie string. Great stuff huh?