Building vagent2 for Varnish Cache 4.0.0 beta 1 for OS X 10.9.2

For those keen bunnies that wishes to jump in and help us test out varnish cache 4.0.0 Beta 1 with varnish-agent 2, here’s how you do it on OS X 10.9.2 Mavericks.

Prerequisites

Homebrew dependencies

Install the following with Homebrew

  • automake 1.14.1
  • libtool 2.4.2
  • pkg-config 0.28
  • pcre 8.34
  • libmicrohttpd 0.9.34

Build varnish cache 4.0.0 beta 1

  1. Download and extract varnish cache 4 https://repo.varnish-cache.org/source/varnish-4.0.0-beta1.tar.gz
  2. run ./autogen.sh
  3. run ./configure
  4. make

Build varnish-agent2 for varnish cache 4.0.0 beta 1

  1. Clone varnish-agent from repo https://github.com/varnish/vagent2
  2. Checkout the varnish-4.0-experimental branch
  3. export VARNISHAPI_CFLAGS=-I/tmp/varnish/varnish-4.0.0-beta1/include
  4. export VARNISHAPI_LIBS="-L/tmp/varnish/varnish-4.0.0-beta1/lib/libvarnishapi/.libs -lvarnishapi"
  5. run ./autogen.sh
  6. run ./configure
  7. make

Note that if you run make install for varnish cache 4 or varnish-agent, it would then install it for you respectively.

Vagrant, Varnish and vmods

Development environment has been plaguing us for a while in my product development department. From dependencies hell to complex setup in operations, our development environment has gone through the usual gauntlet of pains and complaints.

This has changed with Vagrant. It is the single tool that gels the devs with the ops; quintessential devop tool if you will. Not only Vagrant has helped eliminate the “works on my machine” bugs, we use it for automated integration tests. In addition, this one tool has made our development environment setup quick and simple for our HCI guys too.

We do a lot of integration work with Varnish Cache and I thought I would take this opportunity to share this simple Vagrantfile, as an example, to help get started with installing varnish and libdigest-vmod from source.

Note that the provisioning process is rather crude in this example. Rather, the intention here is to out outline the steps required to get varnish and vmods installed and running via Vagrant. For production and future maintainability, do use Chef or Puppet as it can be seamlessly integrated within the Vagrantfile.


# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|

  config.vm.define :varnish do |varnish|
    varnish.vm.box = "varnish"
    varnish.vm.box_url = "http://files.vagrantup.com/precise64.box"
    $script_varnish = <<SCRIPT
echo Installing dependencies, curl
sudo apt-get update
sudo apt-get install curl -y
sudo apt-get install git -y
curl http://repo.varnish-cache.org/debian/GPG-key.txt | sudo apt-key add -
echo "deb http://repo.varnish-cache.org/ubuntu/ precise varnish-3.0" | sudo tee -a /etc/apt/sources.list
echo "deb-src http://repo.varnish-cache.org/ubuntu/ precise varnish-3.0" | sudo tee -a /etc/apt/sources.list
sudo apt-get update
echo ==== Compiling and installing Varnish from source ====
sudo apt-get install build-essential -y
sudo apt-get build-dep varnish -y
apt-get source varnish
cd varnish-3.0.4
./autogen.sh
./configure
make
sudo make install
cd ..
echo done
echo ==== Compiling and installing lib-digest vmod from source ===
git clone https://github.com/varnish/libvmod-digest.git
sudo apt-get install libmhash-dev libmhash2 -y
cd libvmod-digest
./autogen.sh
./configure VARNISHSRC=/home/vagrant/varnish-3.0.4 VMODDIR=/usr/local/lib/varnish/vmods
sudo make install
cd ..
echo ===== done ====
echo ===== firing up varnish via commandline ====
sudo varnishd -a :80 -T :6081 -f /vagrant/test.vcl
touch varnish_vm
SCRIPT

    varnish.vm.provision :shell, :inline => $script_varnish
  end

end

Building on varnish-agent2

The varnish-agent2 existing code base is pretty solid,  rather beautiful I might add. These simple words, keep it simple, has been realised by the following rule of thumb.

– Close to 0 configuration
– “Just works”
– Maintainable
– Generic
– Stateless

For those that are keen to get started on the varnish-agent2 code base, I hope that this document will be of some use. I have used a tiny subset of the Concurrent Object Modeling and architectural design mEThod (COMET), particularly the Requirements Modeling, Analysis Modeling and snippets of the actual Design Model.

Lastly, this document mostly serves as a mental note for myself 🙂

Requirements Modeling

The requirement for the vagent2 is to provide an extendible restful interface for varnish cache. In addition, vagent2 act as the point of integration for varnish cache with other systems – e.g. an administrative or monitoring system.

The use cases are simple right now, and is depicted in the use case diagram below.

vagent2 use cases

vagent2 use cases

Analysis Modeling

vagent2 is designed to support the full spectrum of HTTP method types. A user of the vagent2 will issue these HTTP requests and receive JSON data as response where applicable. Furthermore, the vagent2 is built with modules in mind to address a potentially expanding feature set. Lastly, each modules should be able to communicate and reused by another module.

IPC lies at the heart of the varnish-agent2 code base and message passing is the norm here for implementing an event driven model. Each module follows vagent2’s plugin paradigm, and comes equipped with the module’s own private data set. The plugin therefore follows the IPC set of callback methods, such as ipc_start and ipc_run. These methods are assigned to the appropriate functions within each module.

For the module to be exposed as a restful interface, a simple httpd_register method will hook in the module’s reply method of choice, and expose it appropriately.

For any module, the basic dependencies is depicted below.

Module breakdown

basic module dependencies

Static Modeling

varnish-agent2 ships with a few core modules, such as the logger, shmlog access, httpd and ipc modules. vstatus and vadmin provides access to the shmlog via Varnish API. Note that as of writing this blog, varnish 3.0.3 was used.

These aforementioned modules provide the building blocks for managing varnish cache. For an overview of the static models, see the class diagram below.

Static Model

Static overview of vagent2

Dynamic Modeling

Initialisation
The process of initialising a module is rather straightforward. First, add a new init method for the new module in plugins.h, ensure that you have called the init method in main.c, and of course, allocated some memory for the new plugin in main.c too.

This new module must provide an implementation of the new init method. See diagram below depicting vban’s initialisation process.

vban initialisation process

vban initialisation process

Once initialised, by hooking onto struct agent_plugin* structure, the new module will correspond to the IPC life cycle.

plugin->start  = ipc_start;
plugin->ipc->priv = your_private_plugin_data;
plugin->ipc->cb = your_plugin_callback;

plugin->start is called when your plugin starts. Note that you need to assign a start method if you want the IPC to execute your callback.

plug->ipc->priv refers to a persisted data for your plugin. This can be anything, and as a rule of thumb, this is a good place to hold references to other modules.

plug->ipc-cb refers to the callback method of when ipc_run is issued by another module.

A sample execution path

To tie it all together, the collaboration diagram below illustrate the execution path of issuing a ban to the vagent2. Note that ipc is used to reach the vadmin module.

Issue a ban

Issue a ban