https://github.com/varnish/varnish-devicedetect

It is a VCL set for Varnish which uses regular expressions to group clients into pools like PC, mobile-iphone, mobile-android, tablet-ipad and more based on their User-Agent.

With this you can serve per-device-type content directly from the Varnish cache, without hitting your backend web servers. It is super easy to install with nothing to compile. Just pull the files and include them in default.vcl.

The backend sees the first request like http://example.com/foo.html?devicetype=mobile-android, picks out the GET argument and produces the content that is best suite for small androids. Later requests are cached in Varnish. The main points are: 1) no need to redirect, saving probably a second in page load time since mobile networks are high latency and low bandwidth, 2) increased cache hit rate, which means you don’t have to pay for as many backends. 3) you don’t have to maintain the regular expressions all by yourself.

In addition to the regular expression set and the supporting VCL code to send headers/GET parameter to the backend, you get a system for overriding the detection. Go to /set_ua_device/mobile-android with your usual browser, and later requests will be served as if you had an android phone. Simple as PIE.

The regular expression set will evolve over time (by us, Varnish Software, when a customer asks for it), or by community input. For example, if anyone has a good suggestions on how to differentiate android tablets and android phones, your input is very welcome :)

Munin is an open source solution for collecting metrics from servers and plotting them on trend graphs. It has a simplicity that beats all other monitoring solutions I’ve seen, including super easy plugin creation for adding new charts for your application.

This is for you if you:

• have a couple of  machines / instances here and there (for v1.0), and just want monitoring to work.
• want proper notifications and trending, without doing any configuring yourself.

The idea is that you can spend all your precious hours awake on making your application run supersmooth, instead of tuning monitoring systems.

Pricing looks like will be subscription based with a basic free package (1-3 servers, limited #plugins, but quite usable) and premium package with more nodes and extra features.

We’re in early beta, but have room for some more trial customers.If you want in, drop me an email.

Videre er vinnerbegrepet for de som finner oss via google helt klart klatremus …

En ukes tid etter lansering snakker vi:

• 6200 hits
• 500 besøk
• 200% økning i antall begreper

Klemz til alle som bidrar og har bidratt med å lage den beste norske ordboka, dette blir bra. Tilbakemeldinger og lignende ondsinnet kritikk mottas naturligvis med takk på epost eller TWITAR (@lkarsten).

For de som måtte være ukjent med konseptet handler det om å lage en ordbok basert på slangord. Urbandictionary.com sier de har rundt 5 millioner innslag. Det er jo litt å strekke seg etter, når vi har rundt 100 begreper nå. :-)

planer for fremtiden:

• noe api-greier slik at mobilklientene har noe å snakke med.
• klienter for android/iphone.
• nyhetsbrev med ukas ord?
• kanskje selge noen kaffekopper for å dekke inn hostingutgiftene.

På den tekniske siden bruker vi Django med PostgreSQL i bunn. git til revisjonskontroll, redmine til issuetracking.

Vi trenger litt grafikk, om noen har lyst til å bidra. :-)

tl;dr -versjonen er at det virker omtrent som ISDN, men latensen via satellitt er ganske drøy. Man kommer ut på Internett i Tyskland, og round-trip til Norge er 1-2 sekunder. De har en linuxboks ombord som router IPv4 (ingen IPv6 å se), gjør dns/dhcp og har en transparent webcache (squid by the looks of it).

Som telematiker må det naturligvis undersøkes litt når sjansen bryr seg.

traceroute to www.vg.no (195.88.55.16), 64 hops max, 40 byte packets

1  192.168.33.1 (192.168.33.1)  1 ms  1 ms  1 ms

2  * * *

3  * * *

4  * * *

5  192.168.14.102 (192.168.14.102)  1188 ms  836 ms  961 ms

6  2.239.214.82.in-addr.arpa (82.214.239.2)  721 ms  685 ms  978 ms

7  cr-01-ge0-8.dw.direcpceu.com (62.128.191.190)  1101 ms  958 ms  799 ms

8  pr-02-ge0-1.dw.direcpceu.com (62.128.191.251)  740 ms  748 ms  733 ms

9  pos-2-1-a2.f.core.de.ignite.net (62.134.2.93)  774 ms  689 ms  733 ms

10  t2a7-ge7-0-0.de-fra.eu.bt.net (166.49.172.33)  679 ms  707 ms  714 ms

11  so-7-0-3.edge1.Frankfurt1.Level3.net (212.162.47.109)  685 ms  659 ms  671 ms

12  vlan99.csw4.Frankfurt1.Level3.net (4.68.23.254)  679 ms vlan69.csw1.Frankfurt1.Level3.net (4.68.23.62)  809 ms vlan79.csw2.Frankfurt1.Level3.net (4.68.23.126)  1161 ms

13  ae-72-72.ebr2.Frankfurt1.Level3.net (4.69.140.21)  1490 ms * ae-62-62.ebr2.Frankfurt1.Level3.net (4.69.140.17)  1646 ms

14  ae-48-48.ebr1.Dusseldorf1.Level3.net (4.69.143.177)  901 ms ae-47-47.ebr1.Dusseldorf1.Level3.net (4.69.143.173)  1166 ms ae-48-48.ebr1.Dusseldorf1.Level3.net (4.69.143.177)  1610 ms

15  * * *

16  ae-6-6.car1.Oslo1.Level3.net (4.69.143.225)  1278 ms  1403 ms  1440 ms

17  212.162.27.6 (212.162.27.6)  1361 ms  1306 ms  938 ms

18  te4-2-0.cr2.osls.no.catchbone.net (193.75.3.169) [MPLS: Label 301024 Exp 0]  793 ms  817 ms  790 ms

19  te7-1-0.cr2.xa19.no.catchbone.net (193.75.1.202) [MPLS: Label 308032 Exp 0]  777 ms  970 ms  1420 ms

20  te4-1-0.br1.xa19.no.catchbone.net (193.75.1.50)  890 ms  860 ms  958 ms

21  v4094.m323-rs1.net.kq.no (193.69.44.206)  1119 ms  1629 ms  1230 ms

22  www.vg.no (195.88.55.16)  970 ms  1377 ms  1485 ms

Nedlasting (hm, blir det ikke egentlig _opplasting_ når man er på 30k fot og jobber via en satellittlink?) av en ucachet 10MB stor fil gir 208KB/s.  Litt mistenkelige høyt, montro om målingen min er feil og den faktisk var cachet likevel.

My final method was to, and this may be cheating, look at a pcap or live dump of packets and group per domain the client requested DNS answers from. In essence, utilize that all public IP-over-DNS implementations use a single domain name. The cheating part is because there is no reason for a IP-over-DNS implementation to do this.. I figure the less ethical of IP-over-DNS users can patch this easily.

After collecting n replies from the recursive DNS server to the client and compute the following metrics:

• percentage of replies to the same domain (domain_max_percent)
• bytes per second over the time period it took for n replies to be seen. (bps)
• average number of queries seen per second in the time period. (qps)
• mean packet size in the time period (mps)

The values I got the best results with were:

• mps > 140 bytes
• qps > 2.27
• bps > 560 bytes
• domain_max_percent >= 98%
• n=70 packets

For each sample consisting of n packets from a single client, compute these. If any of the rules above are false, the client is not an IP-over-DNS client. (by this definition)

Early attempts used n=30 packets and mps > 240 bytes, but detection avoidance attempts with extremely low fragment size showed that n=70 packets and mps > 140 byte gave the best results.

Critics may point out that a client may send a lot of fake requests in addition to the IP-over-DNS traffic to a different domain and then skew the max_domain_percent below 98%. Yes, this is possible.

I have Python implementations of all of the methods attempted. I guess realtime detection is the most interesting, and I will put it up on my github account sometime in the near future after cleaning up the code a bit.

In the time since I did my prestudies last spring there seem to have been published a paper on IP-over-DNS detection. It’s on Arxiv, and they use the character distribution in the query strings to match IP-over-DNS clients. This sounds cool, and way better than the Kolmogorovcomplexity attempts that I did.

Om du er litt usikker på denne nymotens webteknologien, gjør som du er vant til og klem det inn samme format som du er vant til. Windows-vinduer er jo best practice på web.

To put it in Wikipedia terms, I believe this blog now contains original research. Crazy.

The setup was to look at packet dumps from a recursive DNS cache, or more precisely, the answers from the local DNS cache to the client. PCAPs were taken at one of the DNS-recursors at NTNU, and also packet dumps of ssh, web and keepalive traffic for nstx, iodine and TUNS were taken.

I was able to within 10-30 seconds detect a client running iodine among university background traffic with no false positives. It was implemented in python with impacket, and works in real time with negligible resource usage.

First of all, I was not able to get the following detection mechanisms to work very well:

• client’s bandwidth usage per time unit. May work if you whitelist your email servers and use 30-60 seconds detection time, but not a very promising method.
• Kolmogorov-complexity. Zip the data parts of the DNS response. The idea was that the complexity (==lower ability to compress it) was higher on IP-over-DNS traffic than on the usual DNS traffic. Didn’t quite work out, but may be feasible with more work.

What I didn’t try, but seemed cool:

• autocorrelation of time between queries. It should be a quite different, since the client is polling constantly.
• time series analysis with wavelets. Complicated math.

I also believe that most people interested in machine learning will find this a very simple task. I did not look into it, but probably should have.

Top tip for people (without training, like me) is to never never never attempt to use uniform sampling for this stuff. I wasted probably a month on figuring this out. You already have a perfect event based sample set, use it for what it is worth.

So, since this blog post is pretty long already, I’m going to save the good stuff for a followup post later this weekend.

Second (or third, fourh .. ) hand, and of course no drivers to be found anywhere.

[15293.788217] usb 4-1: new low speed USB device using uhci_hcd and address 2
[15293.970485] usb 4-1: configuration #1 chosen from 1 choice
[15293.989256] input: TOPRE CO.,Ltd. USB – CardReader         as /devices/pci0000:00/0000:00:1d.2/usb4/4-1/4-1:1.0/input/input16
[15293.989435] generic-usb 0003:0853:0001.0004: input,hidraw0: USB HID v1.00 Keyboard [TOPRE CO.,Ltd. USB - CardReader        ] on usb-0000:00:1d.2-1/input0


No help to be found (nor any contact email addresses??) at Topre’s web pages.  http://www.topre.co.jp/e/products/comp/index.html .

UPDATE:

After contacting Omron Europe, I finally got drivers for it from Cetwin Service which seems to still have support for these. To spare them the extra work I’ve put the driver zip file up on the web here. Since the reader is very old and probably not in sale anywhere any more, this shouldn’t hurt anyones feelings.

In the end my reader was unable to read a single card. It is detected in Windows and seems to work, but any card I try to read it just beeps and flashes unreadable.