Weekly Notes – 26/7/15

Setup.py and rpm.spec:

So recently at work I needed to package a tool we wrote at work. The tool was written in Python (big surprise) and it was designed to be run as a daemon so it had an init-script.

Most Python tools are packaged by writing a `setup.py` script. There are actually a number of tools you can use (I’ve used distutils and setuptools in the past) and there’s a bit of a confusing history regarding which tool is recommended. I mostly use setuptools.

Most (all?) of the software we use in work are packaged as RPMs which are built from a `rpm.spec`. I don’t want to get bogged down in the advantages/disadvantages of this approach here so I’ll keep this part short: It’s more powerful than setup.py scripts but not as well documented. It’s simple enough to install files to specific places that setuptools/distutils don’t make easy (like an init.d script) or to execute code during build/install. On the other hand you really need to understand the ins and outs of your application when writing an rpm.spec. There are some subtle things that happen when you’re building RPMs and that can cause a lot of confusion when you’re starting out.

Now, you can use setuptools to create an RPM package, which is handy. It bypasses the weirdness of rpm.specs and setuptools nicely wraps up some of the messy details of where you should install packages so that they’re available on the python path. Unfortunately, this approach limits you to the features available in setuptools and you lose out on all of rpm.specs flexibility! Just not good enough for my needs.

What I really wanted to do is write both and mix them together. An rpm.spec that delegates some of the hard work to setup.py and looks after the detailed work itself. Luckily this is possible!

Here’s what you need inside your rpm.spec to get setup.py to do the heavy lifting.

The build phase is pretty self explanitory:

%build
python2.7 setup.py build

Then during install you need to point it to install inside the RPM build directory and to save a list of files to INSTALLED_FILES:

%install
python2.7 setup.py install --root=$RPM_BUILD_ROOT --record=INSTALLED_FILES

Then that file list can be passed into the `%files` directive by using `-f`. This means that you won’t need to maintain a list of files that setup.py will be looking after.

%files -f INSTALLED_FILES

You can still list any other files that are taken care of by the rpm.spec after the `%files` directive.

If you try this out yourself, you can check that it all worked by listing the files inside the resulting rpm:

rpm -qpl path/to.rpm

Advertisement

Weekly Notes – 18/7/15

This is the first post in a series that I’ll (hopefully) send out every week. It’ll cover the problems I have during the week as well as how I overcome those issues.

Hopefully this will be of some use to people out there, leave me a comment if you find any of this useful!

Docker machine + docker compose

So I’ve been playing around with docker-machine while working on a pet project of mine: daftpunk.

For those of you that aren’t familiar: docker-machine is a tool to manage hosts running the docker-engine (the server part of docker that actually manages containers). You can use it to create VMs locally using virtual box or vmware that run boot2docker. It can also create instances on various cloud providers and have them all set up and ready to go in minutes. It’s been really handy, I’ve used it to create an EC2 instance to host all the components of my project as containers.

Because daftpunk sports a web front end (written using flask), I thought it would be cool to mount all the flask-related code as a shared volume. That way I could make changes locally and they would be synced to the container hosting the flask app in AWS. Since changes to a flask app would reload the debug server automatically it seemed like a pretty neat way of testing out any changes I was making; at least, in theory. Here’s a mock up of the docker-compose config I’d need to get this to work:

web:
 build: frontend
 volumes:
 - frontend:/opt/frontend
 ports:
 - 5000:5000

It worked just fine when I ran it against my local docker-engine but I spent some time wrestling with the setup when I switched to AWS. The flask server wouldn’t start and when I investigated it seemed that the mount point in the container was empty. Eventually it dawned on me. You just can’t mount files/folders to a remote container.

This makes perfect sense once I stopped to think about it for a minute. What would happen if I turned off my local box? Let’s say for the sake of the argument that the remote container would keep the files it had; those files might be important to it’s operation after all. But things would get complicated if I tried to reconnect to the remote engine. Should it try to find the same stuff on my host and sync with it? This raises questions about merging file changes and that’s really the realm of DVCS‘ like git. Too complicated. Much simpler to just leave that functionality out when working with remote docker-engines.

So what’s the best way to reload the web front end? Well first I needed to add the front end code when I’m building the image instead of mounting it when I’m running the container. In the docker-compose.yaml file I just remove the volumes section and I add this to frontend/Dockerfile:

ADD . /opt/frontend

Then any time I want to test new changes I can use docker-compose to rebuild and redeploy only the front end container, like so:

docker-compose build web
docker-compose up web

This takes a little bit longer but I don’t have to worry about syncing files to remote hosts.

Hello world!

Why I started this blog

I’m not very good at writing about my experiences, I prefer to just talk. In fact I could ramble on about my own thing all night, constantly meandering and exploring tangents as I go.

I’ve decided that I should put more of an effort into collecting my thoughts in blog form so that I can focus them. At the very least I hope this will make me a better developer but maybe I’ll help others who run into the same stumbling blocks on their own journey.

What I intend to write about

To start off I’m hoping to write a weekly summary of interesting things I learned; whether it’s something from my professional work or a pet project.

In the future, I may post updates about some of my pet projects or events I attend but no promises just yet.