Websocket: the road to hell is paved with good intentions

“Server-push technology” was an elusive sales pitch half a decade ago. COMET, Flash and websockets promised ways for servers to “push” messages onto clients. We sold our souls to heavy weight libraries like Icefaces, GWT, or some other JSF abomination that included hixie-75/hixie-76 support. We became oblivious to this devil’s promise and started blaming browsers when stuff did not work. Now with RFC 6455, the realities of websocket for me is still as disjointed as ever. I started off as an evangelist and a staunch supporter for websocket, but as Captain Hindsight would say, “you are doing it wrong”, and I quickly learnt from my mistakes.

This blog is my personal journey with websocket. In the era of devops and cloud (just as elusive and sales pitchy as ever), in my opinion, I find it really hard to see how this protocol would fit elegantly into an elastic, edge-side, micro-service cloud architecture/infrastructure.

tldr;

Websocket does not work out of the box for most load balancers. Furthermore, when you upgrade the connection directly to the backend (see this piece I wrote for Varnish a while back) you lose edge-side load balancing and caching, thus essentially piping the backend directly to the browser one connection per client at the time. Without some additional clever connection multiplexing components in between the load balancer and the websocket server, like websocket-multiplex or some tricked out node-http-proxy, the websocket server will not scale. For those that prefer sales and marketing lingo, this means it is not “cloud enabled” or “web scale.” Furthermore, websocket’s binary protocol, implemented by libraries such as jWebsocket, is extremely hard to debug in production. Unless you are super savvy with Wireshark and regularly eat TCP dumps for breakfast, and not to mention a bit of a masochist, I highly recommend staying away from websocket all together at the time of writing.

Websocket in practice, and asking some hard devops questions

In the past, I have had the displeasure of working with Icefaces and GWT. These Java front end frameworks abstract away the nitty gritty of the network and protocols such as websocket versions, handshakes, messaging format and error handling with elegant MVC models. This is all well and good on the drawing board, but MVC is a design pattern for separation of concerns on the UI level. Not exactly applicable when talking about the complexity and reality of running a websocket server in this internet and mobile driven world.

I have spent past 4 years developing and supporting an application that first utilised websocket. I have to admit, it was fun building directly ontop of jWebsocket, but it was painful and nearly impossible to debug in production. This was alleviated when we went full blown backbone.js, require.js and d3.js. Keeping things simple pays off in the long run. From that experience, I have devised a checklist for any future websocket projects, and help potentially avoiding the same situation from happening again.

• Are there any authentication with this websocket implementation? If so, how does this tie into the user model? (jWebsocket for example requires you to specify a couple of system users for authentication and namespaces/roles for authorisation. These are separated from the existing authentication/authorisation model used by the webapp)
• If it runs within browser, can you tweak the client-side retry logic or connection timeout variables?
• If this runs outside of a browser (and this gets weird fast) are there any TCP client that can be used for automated testing or health checks in production?
• How do you test this automatically? Is it supported by Selenium or phantom.js?
• Can this webserver server be plugged into existing load-balancer? Any additional tweaks and settings required?
• Does this need multiplexing?
• How do you debug this on client-side in production? This is usually not possible because the connection is now elevated into a TCP connection and browsers no longer cares for it.
• How do you debug this on server-side in production? This gets even more tricky as you include multiplexers and load balancers and various other nodes that speaks this binary protocol.
• How do you debug this at all? Not possible if browser gives up the connection and everything in between is a mystery.
• Ok so we can do TCP dump and Wireshark. So are you ready to do TCP dump between somebody’s browser and the origin server and everything else in between?
• Catering to older browsers means Flash fallback. Are you prepared to open an additional Flash port and start supporting this? (and repeat the set of debug and test questions for Flash fallback)
• Does this thing scale? Yes if you multiplex the connection.
• How does it handle reconnection and stateful connection?
• How does the server side handle connections and thread pools handling these logics behind each connection?
• Does the network infrastructure block upgrade headers? Any other firewall rules that might break websocket?
• Lastly, you must be prepared to give up caching.

Benefits with websocket

• True server-push! Yes, this is true. Long-polling or COMET is not exactly “pushing” messages per definition. You are safe from the definition police.
• You get to play with fun new websocket multiplexing code. This is quite cool actually. Mucking around with node.js is always fun. Perhaps you are thinking about building a Varnish VMOD to support websocket multiplexing. Perhaps you are thinking about building some kind of HTTP cacheable stream for websocket messages before stitching them back out as websocket payloads? This is all very exciting indeed!
• Ability to build cool HipChat-like applications