# Directly monitoring Sidekiq in Redis

Another thing that came up recently: we have many (many) sidekiq queues. Each has their own admin interface, but sometimes you just want all of the information in one place. Of course, you could bookmark all of the pages. Or make a single page with a lot of frames (remember HTML frames?). Or use their API. But where’s the fun in that? Instead, let’s dig straight into the redis backend and see what we can see!

First, let’s find the queues. Google + StackOverflow to the rescue1. The queues are (by default), stored in queue:default.

\$ redis-cli

> get queue:default
(nil)


Hmm. There’s a comment on the same post that actually answers that question. It turns out, I have one two three problems:

1. If a queue is empty, the key will not currently exist
2. If you have multiple queues/named queues, they won’t be named default
3. If you used a prefix, that will be part of the queue as well

So… let’s look for a more general solution. It turns out there will always be a list of queues stored at queues or {prefix}:queues. So we can start with:

> keys *queues*
1) "myservice:queues"

> smembers myservice:queues
1) "high"
2) "low"
3) "critical"
4) "medium"

> llen myservice:queue:low
(integer) 2


Nice!

That’s all the information I needed for now. Let’s throw together a quick UI. Now, I really should do this in Ruby, since sidekiq is a Ruby thing, and I’ve started exploring Sinatra as a really light weight server language. But for now, let’s go with what I do best for lightweight services: Flask

I’m going to take what I found about the Redis/Sidekiq earlier and connect to multiple Redii2 and pull out all their data:

redii = {
url: redis.Redis(
host = url.split(':')[0],
port = url.split(':')[1],
decode_responses=True,
)
for url in config.get('redii')
}

def get_counts(host):
logging.info(f'Refreshing data for {host}')

r = redii[host]
results = {'host': host, 'timestamp': time.time()}

try:
total = 0
counts = {}

# Get queue counts
for queue_key in r.keys('*queues'):
prefix = queue_key.split(':')[0] + ':' if ':' in queue_key else ''
for priority in r.smembers(queue_key):
key = f'{prefix}queue:{priority}'
count = r.llen(key)
if count:
counts[priority] = count
total += count

results['counts'] = counts
results['total'] = total

except redis.exceptions.ConnectionError as ex:
results['error'] = str(ex)

return results


Get the *queues, reformat that as a singular to get the keys, get the lengths, and write it all down. Then to be really fancy, let’s make automatic tables:

@app.route('/')
def index():
return tabulate.tabulate(
get_all_counts(),
tablefmt='html',