bcfg2: User-Defined Metadata plugin

I wrote UDMetadata, a bcfg2 Metadata plugin, a few months ago, and have been meaning to release it out into the wild ever since. Here it is, at last!

What is it?

UDMetadata is a Metadata plugin for bcfg2. With UDMetadata, one may define arbitrary data structures on a per-host or per-group basis that are put into the client metadata ‘ud’ attribute. These data structures are used in S/TGenshi templates. The data isn’t limited to just the hostname, groups and bundles in the default client metadata object, and can be referenced more easily than the properties object (which requires calls to the lxml.etree libraries), and can be restricted with <host> and <group> tags, unlike the properties object.

Why is that so great?

As hinted above, the data that can be provided to templates can be much richer than normal metadata, and accessed more easily than properties. In my own environment, where I frequently install new networks with the same types of systems (eg. ‘tomcat-node’, ‘router’, and ‘ldap-server’ profiles) that have similar characteristics (eg. same disk layouts, access the same types of services on the network, similar firewall rules), it can be difficult to install a new network with traditional bcfg2, since the administrator must wade through the various template files making changes that make the new network unique (eg. network numbers, mail relay address, VPN connections). I wanted a way to abstract all such data out of the template files and into a single place where all unique data would reside. Outside of this directory, everything would be the same: same TGenshi and Cfg directories, same Pkgmgr and Rules directories, etc.

So, how does it work?

To show how this works, I will present a simple example and a more complex example.

Simple example

You will rename your Metadata directory to UDMetadata; all the normal Metadata files, like clients.xml and groups.xml, will live there. In this directory, you may also create any number of arbitrarily-named XML files (as long as they don’t conflict with bcfg2’s normal metadata filenames); the UDMetadata plugin will look at all the files in this directory, and if they contain a “UD” tag as the top element, they will be parsed by UDMetadata.

Here is a small example:

<!-- interfaces.xml -->
<UD priority='0'>
  <!-- Network interface configuration -->
  <Interfaces>
    <Group name='mvoffice'>
      <str name='network' value='192.168.1.0'/>
      <str name='netmask' value='255.255.255.0'/>
      <str name='gateway' value='192.168.1.1'/>
      <Host name='alberto.example.com'>
        <str name='ipaddr' value='192.168.1.2'/>
      </Host>
    </Group>
  </Interfaces>
</UD>

This is an overly-simplistic example that could be used to configure all single-homed systems’ network interfaces (let’s pretend we’re not running a UNIX-based router, or the example breaks down!).

As stated, the ‘UD’ tag tells UDMetadata that this is to be parsed and put under the client metadata’s ‘ud’ attribute. You should be familiar with the ‘priority’ attribute; it is the same as that for the Rules or Pkgmgr plugin.

The file may have any number of second-level elements; this file has one, with the ‘Interfaces’ tag. This indicates that the data in this element will be put into metadata.ud.interfaces (the tags are turned into lower-case for the ud object’s attribute names).

Within the Interfaces element, a number of ’string’ data types are defined, and restricted by <Group> and <Host> elements, just as in other bcfg2 plugins. Presently, in addition to the <str> data type, there is also an <eval> data type whose value must be valid python code that will be evaluated. This is extremely dangerous, but very powerful. You will see this in the next example.

When the get_metadata() method is called for the client ‘alberto.example.com’, a dict will be created as metadata.ud.interfaces, with the following structure:

{'ipaddr' : '192.168.1.2',
 'network' : '192.168.1.0',
 'netmask' : '255.255.255.0',
 'gateway' : '192.168.1.1'}

Naturally, this will be passed into S/TGenshi templates in the client metadata object, and we may then use it to write templates such as the following:

# /etc/sysconfig/network-scripts/ifcfg-eth0 for host ${metadata.hostname}
TYPE=Ethernet
DEVICE=eth0
IPADDR=${metadata.ud.interfaces['ipaddr']}
NETWORK=${metadata.ud.interfaces['network']}
NETMASK=${metadata.ud.interfaces['netmask']}
GATEWAY=${metadata.ud.interfaces['gateway']}
ONBOOT=yes

And that’s it! Of course, this example doesn’t represent the real complexities of the files in /etc/sysconfig/network-scripts, which may configure multiple interfaces, bridges, wireless devices, and other things that quickly increase the complexity of the metadata structures and the templates that use them. This is easily possible within this system, though.

More complex example

This example may be downloaded here, a tarball with the example and the UDMetadata.py plugin itself.

I will not explain this example in detail, except for the <eval> tag. In addition to the <str> tag, you may use an <eval> tag that contains python code to be evaluated. I have mostly used this as a quick way to make lists and dicts available to the templates, as in the example. I’m sure that a python hacker could do more interesting or dangerous things with it.

An example of the <eval> tag:

<eval name='myopts' value='{'a':1, 'b':2, 'c':3}' />

This does exactly what you would expect, and might be referenced from a template as follows:

#for key in metadata.ud.options['myopts'].keys()
option ${key} = ${metadata.ud.options['myopts'][key]}
#end

If you look at this example carefully, you’ll see it’s pretty full-featured. In fact, I use the same data in my templates to determine whether the partition should be backed up or reformatted during reinstall (’data’:True), to tell dom0s which LVs to give to which domU (’domU’:'junction.example.com’) as which device (’domUdev’:'xvdb6′), and use the NFSMounts to generate /etc/exports as well.

How to install

This plugin will only work with bcfg2 versions 0.9.6 and higher, since it relies on new configuration file options introduced around then.

1. Download the tarball with example from here.
2. Add the plugin to your /etc/bcfg2.conf:

   [server]
   plugins = UDMetadata

3. Copy UDMetadata.py to your bcfg2 tree. On my system:

   # cp UDMetadata.py /usr/lib/python2.4/site-packages/Bcfg2/Server/Plugins

4. The files that normally belong in your Metadata directory (clients.xml, groups.xml, etc.) now belong in your UDMetadata directory. However, there are some things (I don’t recall what right now) that still expect these files to be in Metadata. The easiest thing to do is to make a symlink:

   ln -s Metadata /var/lib/UDMetadata

5. Create your UD XML files in the UDMetadata directory, and adapt your templates to use them.

You’re finished!

Other things to note

First, I am not a python programmer, and I do not understand the inner workings of bcfg2.
This is my first python work of more than just a few lines. It’s probably buggy, and may erase all your files. All other conceivable disclaimers apply.

That said, I have used this plugin, plus a few other simple modifications to bcfg2, to concentrate all network-specific configuration data into several files in the UDMetadata directory. I have some fairly complicated templates that take care of machine configuration from even before the time it is first installed. Bcfg2 manages the PXE servers and their hosts’ configuration files, the Xen dom0’s domU configuration files, the kickstart server’s ks.cfg files for each host, as well as all networking, firewalling, clustering and other configuration. When it is time to add a new host of an existing profile, it is usually just a matter of entering its IP address, and which dom0 and LVM partition it will live on, and then all configuration is taken care of from kickstart to production configuration.

I am very eager to hear the opinions of those more experienced with bcfg2. Thank you for taking the time to comment!