Counting and Sizing S3 Buckets

A long time ago in a galaxy far far away, I wrote up a script that I used to take an AWS S3 bucket and count how many objects there were in the bucket and calculate its total size. While you could get some of this information from billing reports, there just wasn’t a good way to get it other than that at the time. The only way you could do it was to… iterate through the entire bucket, summing as you go. If you have buckets with millions (or more) objects, this could take a while.

Basically:

conn = boto.connect_s3()
for bucket in sorted(conn.get_all_buckets()):
    try:
        total_count = 0
        total_size = 0
        start = datetime.datetime.now()

        for key in bucket.list_versions():
            # Skip deleted files
            if isinstance(key, boto.s3.deletemarker.DeleteMarker):
                continue

            size = key.size
            total_count += 1
            total_size += size

        print('-- {count} files, {size}, {time} to calculate'.format(
            count = total_count,
            size = humanize.naturalsize(total_size),
            time = humanize.naturaltime(datetime.datetime.now() - start).replace(' ago', '')
        ))


The Fires of Heaven

The Fires of Heaven is mostly a book to extend what we’ve seen before.

Elida solidifies her control on the tower while the rebels come together to oppose her. Nynaeve tugs her braid. A lot. Elayne and Nynaeve end up in the circus (not even kidding) and with a captured Forsaken...


Creating a temporary SMTP server to 'catch' domain validation emails

One problem that has come up a time or two is dealing with email-based domain validation (specifically in this case for the issuance of TLS certificates) on domains that aren’t actually configured to receive email. Yes, in a perfect world, it would be easier to switch to DNS-based validation (since we have to have control of the DNS for the domain, we need it later), but let’s just assume that’s not an option. So, how do we ‘catch’ the activation email so we can prove we can receive email on that domain?


The Blood Mirror

Look at your mistakes long enough to learn from them, then put them behind you.

On one hand, The Blood Mirror takes the storyline of the previous books and turns it up even more.


The Forest

The Forest came out of /r/WritingPrompts:

[WP] Instead of Oceans, they are all big forests, that gets taller and darker instead of deeper, with more dangerous animals living further out in the forest. A person decides to cross the Mariana Trench


Black Water

I tried. Really I did. But when you read as quickly as I do and realize that you’ve already spent more than a month on one book, it’s time to move on.

I think most of my frustration with this book comes from the fact that Pendragon is a teenager and acts like it. He is hit in the face with a world where humanoid cats are the dominant species and humans are little more than animals and just cannot get it through his head. Over and over, he expects the world to work one way, despite all evidence to the contrary.


The Shadow Rising

Mat snorted. “I don’t want to be any bloody hero.”

Bit late for that. For all of them.


The Great Hunt

Amusingly, due to the library not having a copy of The Eye of the World , the first two times I read the series, I actually started with The Great Hunt . While the Eye of the World could arguably stand alone (albeit with a somewhat unsatisfying feeling, given the sheer quantity of untapped potential in the world building), it’s in the Great Hunt that things really start getting crazy.

Rand is just now starting to understand what being the Dragon Reborn could mean. Perrin has a connection to the wolves, something very old, come new again. Mat is still cursed by his dagger and yet he's the one to finally sound the Horn . You have the first hints of parallel universes and world hopping, along with the World of Dreams–which will turn out to be rather important to the universe all together–along with a few more Forsaken coming out of the woodwork. Lanfear… Lanfear proves that being loved by a power hungry hundred/thousand year old super powerful jealous woman can be… interesting.