NMEA2000 and CANbus

I’ve been looking at NMEA2000 and CANbus lately. I’m attempting to reverse engineer the binary protocol seen coming out of the USB port on a Garmin GND10 network bridge, and that opens a whole field of new and interesting stuff to know about.

Quick background: NMEA2000 is a application-level protocol that is used on ships and smaller boats, to communicate things like heading, speed and position between sensors (gps, compass, etc) and displays on the bridge. The physical, datalink and network layers (to the extent that it applies) in this are CANbus.

It is truly fascinating how complicated “industry standard” can be. Adding to that, each of the 8 ISO1939 standards defining it is 168CHF each. Not an option for me at the moment, so here I go digging in binary data. (And in retrospect, complain less about RFCs in the future. They are free, open and mostly well thought out.)

The Wikipedia nodes for both these contain lots of facts, but perhaps doesn’t give you the overview you need as an implementor.

CANbus can be sent over many different physical setups, for NMEA2000 there are five wires: ground, +12V vcc, can-hi, can-lo and shield. Typically you have a canbus transciever circuit/IC to do the differential encoding on can-lo and can-hi, and a separate canbus controller for the protocol itself. The connectors for NMEA2000 is a round quick-release kind, but you can also do canbus over cheap cat5+rj45 if you want. (250kbit/s is rather forgiving these days)

CANbus frames have a 32 bit identifier and up to 8 bytes (yes, this is quite adorable) of payload. The wire format contains additional bits here and there, a length field and a CRC and DC-correcting tailer. All this is probably interesting if you create the controllers, but for anyone writing software it doesn’t matter. Especially the part about the frame identifier is either 11 bit or 29 bits, you can ignore. At least for NMEA2000 only CAN2.0B with 29 bit identifiers are used.

On the software side, you get: 1) 32bit frame identifier, 2) the length indicator, and 3) the >=8 bytes of data. Everything else is handled by your interfacing hardware. Hurray.

The 32 bit identifier is a bitfield consisting of source, destination, priority and the Parameter Group Number (PGN).

In NMEA2000 most frames are broadcasted without any specific receivers in mind. The PGNs are defined by the non-public standard made by NMEA, and defines the number of payload bits and what they contain.

Which range a PGN is in also define which framing method is used. CAN frames with a PGN in a single-frame range has the single-frame 8 bytes of payload. Some PGNs are in a “fast-packet” range, where a collection of (up to) 32 CAN-frames can be sent with a session id, sequence counter embedded in the first 8 bits of each payload. The first frame in the sequence additionally has a sequence length in the second byte. This allows the receiver to reassemble the sequence in userspace for a combined payload of up to 233 bytes.

I have no idea how controllers handle lost frames in a fastpacket sequence, most likely they just drop the whole sequence. I also wonder if anything here is complicated enough to that out-of-order delivery can happen. Haven’t seen it yet, though.

There is an additional “ISO” transfer mode that is a (“unicast”) request/response method. Back when I was playing with ODB2 in cars, you had to request a specific ID with data. I think this what was being used there, but haven’t verified. It supports up to 1785 bytes, so obviously there is some segmentation and reassembly method there as well. I haven’t looked at it yet, as I’m mostly interested in the broadcasted updates.

On the network management side, there is an ARP/ND alike way for connecting units to claim source IDs. Lower source id means higher network priority on collisions. so there is some magic to it. I haven’t come this far yet.

Trying to keep this short and readable, let me round off with mentioning the CANbus network interface in Linux. There is kernel support now and you get a network interface with a whopping 8 byte MTU. There is even a loopback interface for it, so you can talk CAN between applications if you don’t have a physical CAN interface. Look at the documentation for can-utils.git for details.

I’ll try to post again when I figure out more of this stuff.

Sources:

 

Posted in stuff | Tagged , , , , , | Leave a comment

Shutting down hostedmunin.com

Keeping this short: I’m shutting down my side project running a hosted Munin monitoring service from mid August 2016.

Priorities have changed, and since I don’t run many servers these days I don’t spend the time maintaining it that such a service deserves.

I hope it has been useful service. I know I’ve learned a lot from this experience.

-Lasse

Posted in Uncategorized | Leave a comment

Varnish Device detection looking for maintainer

We’re looking to have someone take over maintenance of the Varnish Device detection VCL set.

The last couple of years we haven’t really served it right, and it is time to let someone with more practical/hands-on experience take over.

I’ve written an explanation in the README.rst file. Short version is that we’re good at fixing Varnish crashes and configuring it. Expert knowledge about User-Agents and what is a reasonable way to group 2015-clients we don’t really have.

Compensation is the usual in open source community projects, minor fame from your peers and some contributor gear. T-shirts and Varnish Cache coffee mugs in abundance shall be yours.

So, if you’re up for it, please drop me an email and we’ll see what we can figure out.

 

Posted in varnish | Tagged , | Leave a comment

Report from Varnish Developer meeting 2015-12-04

 

20151204_123624

View of Rotterdam Centraal from floorplanner’s offices.

Last Friday the Varnish Cache team met up in Rotterdam to discuss all things Varnish and to work together on ongoing tasks.

There is work being done on merging UPLEX’s consistent hashing backend director into Varnish. After a delightful discussion I believe Nils decided it should be called the shard (sharding) director.

Dynamic backend definition support is moving along and Dridi showed off work done on his new DNS based director.

Further on the load balancing side of things, Arianna is looking at gradual traffic rampup when a backend comes back from being sick.

Guillaume and Dag dug deep into the proposed specification for describing HTTP/2 in varnishtest test cases. This is a prerequisite for a robust HTTP/2 implementation in Varnish.

Github transition is moving along, with migration scripts and work from trac being worked on. It may be a while until Github’s API rate limiting lets our hosts do any requests again, though.

Martin and Poul-Henning spent some time discussing the object storage API to better support the upcoming persistent storage module being made by Varnish Software these days.

20151204_155149

Varnish developers hard at work

Thanks to Floorplanner.com for allowing us to use their facilities in Rotterdam, and to Phil from Dyn for setting this up.

Posted in varnish | Tagged , | Leave a comment

Scanning for multicast TV channels

In recent times, I’ve cut the cord and stopped watching TV from my local cable company.

This gave the need to look for IPv4 multicast groups that had active talkers, so I (on occasion) could stream some live TV over the internet instead. (don’t overthink the logic here .. ;))

The Norwegian research network has some free/open TV channels (NRK, TV2, Frikanalen), and I was thinking perhaps there are more available further out?

IPv4 multicast has a pretty small address space. 244/8 and up to the top. Shouldn’t be to hard to join all of them and see if something comes down the pipe?

Steps:

1) cook up a libpcap listener that reports destination ip and port of any multicast that is coming in, as well as bitrate. you probably want to filter mdns, sap, upnp and some other ports to keep the false positives low. I’ve made gist with the one I wrote here.

2) Join every multicast group under the sun, one by one. Just bind a socket to it, the port doesn’t matter. Rinse and repeat. I spent 10 seconds in each group, and joined about 50 at the time. I recommend that you go buy your network operations people some beer in advance, as IGMP implementations seem to fall over after a few days of heavy joins/parts. I recommend perhaps 5 or 10 concurrent groups. Code is here.

Some results, either from direct scanning or from scanning subnetworks that were sending SAP announcements:

  • There are some TV channels on 233.0.59.0/24. Deutsche Welle and NHK World in 1080p.
  • The Hellenic Parliament [sic] TV is on 233.21.32.200:1234.
  • There are quite a few surveillance cameras hooked up, ready for your viewing pleasure!
  • SVT1 is still around in Uninett.
  • Some weird music channel called mtvU. Endless commercials, so probably american? :)

There are some more findings, but I don’t think the owners would appreciate publicity. I’m pretty sure there is more out there, if someone cares to take another look.

Anyway, I’ve had the code laying around for a while. Perhaps someone will find it useful. It worked well enough for me.

Posted in Uncategorized | Leave a comment

Introducing hitch – a scalable TLS terminating proxy.

The last couple of weeks we’ve been pretty busy making SSL/TLS support for Varnish Cache Plus 4. Now that the news is out, I can follow up with some notes here.

The setup will be a TLS terminating proxy in front, speaking PROXY protocol to Varnish. Backend/origin support for SSL/TLS has been added, so VCP can now talk encrypted to your backends.

On the client-facing side we are forking the abandoned TLS proxy called stud, and giving a new name: hitch.

hitch will live on github as a standalone open source project, and we are happy to review patches/pull requests made by the community. Here is the source code: https://github.com/varnish/hitch

We’ve picked all the important patches from the flora of forks, and merged it all into a hopefully stable tool. Some of the new stuff includes: TLS1.1, TLS1.2, SNI, wildcard certs, multiple listening sockets. See the CHANGES.rst file updates.

Varnish Software will provide support on it for commercial uses, under the current Varnish Plus product package.

Posted in varnish | Tagged , , | Leave a comment

PROXY protocol in Varnish

Dag has been working implementing support for HAProxy’s PROXY protocol[1] in Varnish. This is a protocol adds a small header on each incoming TCP connection that describes who the real client is, added by (for example) an SSL terminating process. (since srcip is the terminating proxy)

We’re aiming for merging this into Varnish master (so perhaps in 4.1?) when it is ready.

The code is still somewhat unfinished, timeouts are lacking and some polishing needed, but it works and can be played with in a development setup.

Code can be found here: https://github.com/daghf/varnish-cache/tree/PROXY

I think Dag is using haproxy to test it with. I’ve run it with stunnel (some connection:close issues to figure out still), and I’d love if someone could test it with ELB, stud or other PROXY implementations.

1: http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt

Posted in varnish | Tagged , , , , , | Leave a comment

Varnish VMOD static code analysis

I recently went looking for something similar to pep8/pylint when writing Varnish VMODs, and ended up with OCLint.

I can’t really speak to how good it is, but it catches the basic stuff I was interested in.

The documentation is mostly for cmake, so I’ll give a small tutorial for automake:

  • (download+install oclint to somewhere in $PATH)
  • apt-get install bear
  • cd libvmod-xxx
  • ./autogen.sh; ./configure –prefix=/usr
  • bear make # “build ear” == bear. writes compile_commands.json
  • cd src
  • oclint libvmod-xxx.c # profit
  • Which will tell you about unused variables, useless parentheses, dead code and so on.

    Posted in varnish | Tagged , , , | Leave a comment

    Announcing libvmod-tcp: Adjust Varnish congestion control algorithm.

    I’ve uploaded my new TCP VMOD for Varnish 4 to github, you can find it here:
    http://github.com/lkarsten/libvmod-tcp.

    This VMOD allows you to get the estimated client socket round trip time, and then let you change the TCP connection’s congestion control algorithm if you’re so inclined.

    Research[tm][0] says that Hybla is better for long high latency links, so currently that is what it is used for.

    Here is a quick VCL example:

    if (tcp.get_estimated_rtt() > 300) {
    set req.http.x-tcp = tcp.congestion_algorithm("hybla");
    }

    One thing to note is that VCL handling is very early in the TCP connection lifetime. We’ve only just read and acked the HTTP request. The readings may be off, I’m analyzing this currently.
    (As I understand it the Linux kernel will keep per-ip statistics, so for subsequent requests this should get better and better..)

    References:
    0: Esterhuizen, A., and A. E. Krzesinski. “TCP Congestion Control Comparison.” (2012).

    Posted in varnish | Tagged , , , | Leave a comment

    Fresh Varnish packages for Debian/Ubuntu and Redhat systems

    We use continuous integration when developing Varnish Cache. This means that we run our internal test suite (varnishtest) on all commits, so we catch our mistakes earlier.

    This pipeline of build jobs sometimes end up with binary packages of Varnish, which may be useful to people when they know they exist. They may not be the easiest to find, which this blog post tries to remedy.

    Development wise, Varnish Cache is developed with GIT with a master branch for development and a set of production branches, currently 3.0 and 4.0.

    Unreleased packages for Varnish master can be found here: https://jenkins.varnish-software.com/view/varnish-master/

    Unreleased packages of Varnish 4.0 can be found here: https://jenkins.varnish-software.com/view/varnish-4.0/

    (There is also a set of 3.0 jobs, but you should really go for 4.0 these days.)

    The latest commits in each of the production branches may contain fixes we’ve added after the last production release, but haven’t cut a formal release for yet. (For example there are some gzip fixes in the 3.0 branch awaiting a 3.0.6 release, which I really should get out soon.)

    Some jobs in the job listing just check that Varnish builds, without creating any output (or artifacts as Jenkins calls it.) This applies for any jobs with “-build-” in the name, for example varnish-4.0-build-el7-x86_64 and varnish-4.0-build-freebsd10-amd64.

    The Debian and Ubuntu packages are all built from one job currently, called varnish-VERSION-deb-debian-wheezy-amd64. Press “Expand all” under artifacts to get the full list.

    Redhat/RHEL packages are built in the different el5/el6/el7 jobs.

    The unreleased packages built for 3.0 and 4.0 are safe. This is the process used to build the officially released packages, just a step earlier in the process. The varnish-master packages are of course failing from time to time, but that is to be expected.

    The version numbers in the packages produced may be a bit strange, but that is what you get with unreleased software builds.

    I’m happy to improve this process and system if it can help you run never versions of Varnish, comments (either here or on IRC) are appreciated.

    Posted in stuff, varnish | Tagged , , | 6 Comments