Observation Server

For a number of years now, I’ve been writing down my ‘observations’. Essentially, it’s a semi-structured set of text files that I keep in Dropbox. One for each day, in a folder by month. I record interesting people I see, things I did worth doing, and things my children did which were adorable.

After a while, I started wanting to look back, so first, I wrote a relatively simple script that would go back through my archives and send me everything I did 1/2/3/4/etc years ago. That worked well enough, but it ended up generating a lot of emails to go through some days. So the second generation is a server that can format those pages and display them as a nice webpage.

The most interesting part perhaps was dealing with the tarballs that I keep the archives in (they’re plain text, so they compress very well). I wanted to keep them compressed, so I had to decompress them in memory on the fly.

read more...


Adventures in Racket: gzip

In my research, I work with a lot of rather large text files–on the order of gigabytes if not terabytes per file. Since they’re plain text, they’re generally rather compressible though, so it makes sense to gzip them while they’re on disk. The drawback though comes when you’re working with them. There are a few options though.

read more...