Today’s goal: Turn a collection of YouTube links into a podcast.

Start with a config.json like this:

{
  "brandon-sanderson": [
    "https://www.youtube.com/watch?v=H4lWbkERlxo&list=PLSH_xM-KC3ZuOZayK68JAAjj5W9ShnFVC",
    "https://www.youtube.com/watch?v=YyaC7NmPsc0&list=PLSH_xM-KC3ZtjKTR2z8rPWxv1pP6bOVzZ",
    "https://www.youtube.com/watch?v=o3V0Zok_kT0&list=PLSH_xM-KC3ZuteHw3G1ZrCDWQrAVgO0ER"
  ]
}

And it will automatically download all referenced YouTube videos, convert them to MP3 (both using youtube-dl), and serve an RSS feed that’s compatible with most podcast programs.

Tested URLs include:

• Playlist URLs (like the above)
• Single video URLs
• Channel URLs

Most youtube URLs should work though.

read more...

Another quick post.

What feels like a lifetime ago, I wrote a post about finding ec2 instances by name. I honestly use that script just about every day, mostly for automatically finding instances to SSH to (a la SSH config tricks). But there are a few other quick things I’ve done with it:

• ec2-script - Run a script on all instances of a given name
• ec2-disk - A specialization of ec2-script to check main disk usage
• terminate - A script that I use with ec2 to terminate instances from the command line
• ec2-cycle - Slow cycle a given set of ec2 instances by terminating so many per minute

All of which are included in my dotfiles.

read more...

The Sender Policy Framework is one of those things that’s really powerful and useful to help prevent phishing and email spam, but can be a royal pain to work with. Specifically, SPF is a series of DNS TXT records¹ with a specific format that can be looked up by any email service to verify that an email was sent by a server that should be authorized to send email on your behalf. For example

"v=spf1 ip4:192.0.2.0/24 ip4:198.51.100.123 a -all"

• v=spf1 - tells the client this is an SPF record and should always start the record
• {key}[:{value}]? - one of many different key/value pairs that can define the record
  • in the case above a ip4 key species an IPv4 address range that can send emails on your behalf (the value can be optional)
  • the a above is another special case where if the sender domain ([email protected] would be example.com) resolves via a DNS A record to the server that sent the email, it’s allows
• -all is a fallthrough case meaning ‘fail all that didn’t match a previous case

There are a number of other cases, but we’ll get to the other interesting ones in a bit.

read more...

Another thing that came up recently: we have many (many) sidekiq queues. Each has their own admin interface, but sometimes you just want all of the information in one place. Of course, you could bookmark all of the pages. Or make a single page with a lot of frames (remember HTML frames?). Or use their API. But where’s the fun in that? Instead, let’s dig straight into the redis backend and see what we can see!

read more...

One thing that I often need to do is deal with a large collection of database servers in different clusters and in different environments. On top of that, sometimes, I want a UI, sometime I want a CLI to script. And sometimes I’m on a VPN and sometimes I’m not. All together, it’s a rather complicated number of saved connections and CLI switches and everything else. All together, I want:

• Specify the cluster, environment, and mode (read/write/adhoc)
• Specify if I want to run via CLI or via UI
• Specify an optional user with safely stored and used passwords
• Automatically connected via SSH tunnel if I’m not on VPN, but not if I am (for CLI or VPN)

Let’s do it!

read more...

For a number of years now, I’ve been writing down my ‘observations’. Essentially, it’s a semi-structured set of text files that I keep in Dropbox. One for each day, in a folder by month. I record interesting people I see, things I did worth doing, and things my children did which were adorable.

After a while, I started wanting to look back, so first, I wrote a relatively simple script that would go back through my archives and send me everything I did 1/2/3/4/etc years ago. That worked well enough, but it ended up generating a lot of emails to go through some days. So the second generation is a server that can format those pages and display them as a nice webpage.

The most interesting part perhaps was dealing with the tarballs that I keep the archives in (they’re plain text, so they compress very well). I wanted to keep them compressed, so I had to decompress them in memory on the fly.

read more...

If you have a website that allows users to submit URLs, one of the (many many) things people will try to do to break your site is to submit URLs that use the javascript: protocol (rather than the more expected http: or https:). This is almost never something that you want, since it allows users to submit essentially arbitrary code that other users will run on click in the context of your domain (same origin policy).

So how do you fix it?

First thought would be to try to check the protocol:

> safe_url = (url) => !url.match(/^javascript:/)
[Function: safe_url]

> safe_url('http://www.example.com')
true

> safe_url('javascript:alert(1)')
false

read more...

Quite often, I’ll find myself wanting to query and manipulate MySQL data entirely on the command line. I could be building up a pipeline or working on a task that I’m going to eventually automate but haven’t quite gotten to yet. Whenver I have to do something like that, I have a small pile of scripts I’ve written over time that help out:

• skiphead: Skip the first line of output, used to skip over headers in a query response
• skipuntil: Skip all lines until we see one matching a pattern, used to resume partial tasks
• commaify: Take a list of single values on the command line and turn them into a comma separated list (for use in IN clauses)
• csv2json: a previously posted script for converting csv/tab delimited output to json
• jq: not my script, but used to take the output of csv2json and query it further in ways that would be complicated to do with SQL

Admitedly, the first two of those are one liners and I could easily remember them, but the advantage of a single command that does it is tab completion. sk<tab>, arrow to select which one I want, and off we go. I could put them as an alias, but I don’t always use the same shell (mostly fish, but sometimes Bash or Zsh).

read more...

Today I found the need to look through all old versions of a file in S3 that had versioning turned on. You can do it through the AWS Console, but I prefer command line tools. You can do it with awscli, but the flags are long and I can never quite remember them. So let’s write up a quick script using boto3 (and as a bonus, try out click)!

read more...

Source: No Matter How You Slice It

Part 1: Given a list of overlapping regions defined by (left, top, width, height) count how many integer points occur in more than one region.

read more...