Advent of Code 2023 - Testing and Timing

A quick follow up to Advent of Code 2023: testing and timing.

It’s been bothering me a bit that I haven’t had a generic way to run tests and timing on every problem as I’m going.

So let’s fix it!


I’ve been filling out test data with a theoretical future macro like this:

// #[aoc_test("data/test/01.txt", 142)]
// #[aoc_test("data/test/01b.txt", 209)]
// #[aoc_test("data/01.txt", 53651)]

Which almost works (and would be fine if we were using a procedural macro). But if we want to use a more simple declarative macro with macro_rules! instead, we have to deal with macro hygiene. Basically, we can’t create new names (which we’d love to do for the module names).

So instead, I ended up with this:

aoc_test::generate!{day01_part1_test_01 as "test/01.txt" => "142"}
aoc_test::generate!{day01_part1_test_01b as "test/01b.txt" => "209"}
aoc_test::generate!{day01_part1_01 as "01.txt" => "53651"}

A bit longer than I’d like, but workable. And with a bit of regex (and a script to actually get the package and binary name), we can automatically rewrite the old form to this.

The actual macro:

macro_rules! generate {
    ($name:tt as $input_file:tt => $expected:tt) => {
        mod $name {
            use super::*;
            use std::fs::OpenOptions;
            use std::io::prelude::*;

            fn run_test() {
                let filename = format!("../../data/{}", $input_file);
                let input = std::fs::read_to_string(filename).unwrap();
                let actual = process(input.as_str()).unwrap();

                assert_eq!(actual, $expected);

That’s really it! Running it with cargo test we get a giant pile of output. cargo nextest does much better:

$ cargo nextest run --release

warning: crate `day__day__` should have a snake case name
  = help: convert the identifier to snake case: `day_day`
  = note: `#[warn(non_snake_case)]` on by default

warning: `day__day__` (lib) generated 1 warning
warning: `day__day__` (lib test) generated 1 warning (1 duplicate)
    Finished release [optimized] target(s) in 0.05s
    Starting 182 tests across 119 binaries
        PASS [   0.007s] day01::bin/part1 day01_part1_test_01b::run_test
        PASS [   0.005s] day01::bin/part1-first-last day01_part1_first_last_01::run_test
        PASS [   0.009s] day01::bin/part1 day01_part1_test_01::run_test
        PASS [   0.012s] day01::bin/part1 day01_part1_01::run_test
        PASS [   0.005s] day01::bin/part2 day01_part2_01::run_test
        PASS [   0.006s] day01::bin/part1-first-last tests::test_first_and_last
        PASS [   0.006s] day01::bin/part1-first-last day01_part1_first_last_test_01::run_test
        PASS [   0.008s] day01::bin/part2 day01_part2_test_01::run_test
        PASS [   0.007s] day02 parse::tests::test_games
        PASS [   0.007s] day02 parse::tests::test_game
        PASS [   0.008s] day02 parse::tests::test_color
        PASS [   0.008s] day01::bin/part2 day01_part2_test_01b::run_test
        PASS [   0.005s] day02 parse::tests::test_round
        PASS [   0.005s] day02::bin/part1 day02_part1_02::run_test
        PASS [   0.005s] day02::bin/part1 day02_part1_test_02::run_test
        PASS [   0.006s] day02::bin/part2 day02_part2_02::run_test
        PASS [   0.006s] day02::bin/part2 day02_part2_test_02::run_test
        PASS [   0.008s] day03 types::tests::test_neighbor
        PASS [   0.006s] day03 types::tests::test_parse_longer_number
        PASS [   0.007s] day03 types::tests::test_parse_numer
        PASS [   0.007s] day03::bin/part2 day03_part2_03::run_test
        PASS [   0.007s] day03::bin/part2 day03_part2_test_03::run_test
        PASS [   0.008s] day03 types::tests::test_symbols
        PASS [   0.007s] day04 parse::test::test_parse_card_whitepsace
        PASS [   0.007s] day04 parse::test::test_parse_card
        PASS [   0.010s] day03::bin/part1 day03_part1_03::run_test
        PASS [   0.010s] day03::bin/part1 day03_part1_test_03::run_test
        PASS [   0.006s] day04::bin/part2 day04_part2_04::run_test
        PASS [   0.007s] day04::bin/part1 day04_part1_test_04::run_test
        PASS [   0.008s] day04::bin/part1 day04_part1_04::run_test
        PASS [   0.008s] day04::bin/part2 day04_part2_test_04::run_test
        PASS [   0.008s] day05 types::test::test_category_map_apply
        PASS [   0.006s] day05::bin/part1 day05_part1_05::run_test
        PASS [   0.006s] day05::bin/part1 day05_part1_test_05::run_test
        PASS [   0.007s] day05 types::test::test_range_map_apply
        PASS [   0.005s] day05::bin/part2 day05_part2_test_05::run_test
        PASS [   0.008s] day05 types::test::test_range_map_range_apply
        PASS [   0.006s] day05::bin/part2 day05_part2_05::run_test
        PASS [   0.007s] day05::bin/part2-brute day05_part2_brute_test_05::run_test
        PASS [   0.006s] day06::bin/part1 day06_part1_06::run_test
        PASS [   0.006s] day06::bin/part1 day06_part1_test_06::run_test
        PASS [   0.005s] day06::bin/part1-brute day06_part1_brute_test_06::run_test
        PASS [   0.007s] day06::bin/part1-brute day06_part1_brute_06::run_test
        PASS [   0.007s] day06::bin/part2 day06_part2_06::run_test
        PASS [   0.005s] day06::bin/part2-brute day06_part2_brute_test_06::run_test
        PASS [   0.006s] day06::bin/part2 day06_part2_test_06::run_test
        PASS [   0.005s] day07::bin/part2 day07_part2_test_07::run_test
        PASS [   0.013s] day05::bin/part2-brute-par day05_part2_brute_par_test_05::run_test
        PASS [   0.007s] day08::bin/part1 day08_part1_08::run_test
        PASS [   0.009s] day07::bin/part1 day07_part1_test_07::run_test
        PASS [   0.005s] day08::bin/part2 day08_part2_test_08c::run_test
        PASS [   0.011s] day07::bin/part1 day07_part1_07::run_test
        PASS [   0.009s] day08::bin/part1 day08_part1_test_08::run_test
        PASS [   0.008s] day08::bin/part2 day08_part2_test_08::run_test
        PASS [   0.011s] day07::bin/part2 day07_part2_07::run_test
        PASS [   0.007s] day08::bin/part2 day08_part2_test_08b::run_test
        PASS [   0.006s] day08::bin/part2-brute day08_part2_brute_test_08c::run_test
        PASS [   0.008s] day08::bin/part2-brute day08_part2_brute_test_08b::run_test
        PASS [   0.009s] day08::bin/part2-brute day08_part2_brute_test_08::run_test
        PASS [   0.008s] day09::bin/part1 day09_part1_09::run_test
        PASS [   0.006s] day09::bin/part1 day09_part1_test_09::run_test
        PASS [   0.006s] day09::bin/part2 day09_part2_09::run_test
        PASS [   0.006s] day10::bin/part1 day10_part1_test_10b::run_test
        PASS [   0.006s] day09::bin/part2 day09_part2_test_09::run_test
        PASS [   0.007s] day10::bin/part1 day10_part1_test_10::run_test
        PASS [   0.007s] day10::bin/part1 day10_part1_test_10c::run_test
        PASS [   0.007s] day10::bin/part2 day10_part2_test_10b::run_test
        PASS [   0.009s] day10::bin/part2 day10_part2_test_10::run_test
        PASS [   0.007s] day10::bin/part2 day10_part2_test_10e::run_test
        PASS [   0.008s] day10::bin/part2 day10_part2_test_10c::run_test
        PASS [   0.006s] day10::bin/part2 day10_part2_test_10f::run_test
        PASS [   0.027s] day06::bin/part2-brute day06_part2_brute_06::run_test
        PASS [   0.008s] day10::bin/part2 day10_part2_test_10d::run_test
        PASS [   0.008s] day11::bin/part1 day11_part1_11::run_test
        PASS [   0.007s] day11::bin/part1 day11_part1_test_11::run_test
        PASS [   0.006s] day11::bin/part2 day11_part2_test_11::run_test
        PASS [   0.007s] day12::bin/part1 day12_part1_test_12::run_test
        PASS [   0.008s] day11::bin/part2 day11_part2_11::run_test
        PASS [   0.006s] day12::bin/part1-orig day12_part1_orig_test_12::run_test
        PASS [   0.009s] day12::bin/part2 day12_part2_test_12::run_test
        PASS [   0.009s] day12::bin/part2 tests::hash_question
        PASS [   0.006s] day12::bin/part2 tests::simple_questions
        PASS [   0.009s] day12::bin/part2 tests::no_questions
        PASS [   0.039s] day08::bin/part2 day08_part2_08::run_test
        PASS [   0.007s] day13 types::bounds_test::test_contains
        PASS [   0.009s] day13 types::bounds_test::test_include
        PASS [   0.011s] day12::bin/part2-typed day12_part2_typed_test_12::run_test
        PASS [   0.010s] day13 types::point_test::test_reflect_x
        PASS [   0.009s] day13::bin/part1 day13_part1_13::run_test
        PASS [   0.012s] day13 types::point_test::test_reflect_y
        PASS [   0.009s] day13::bin/part1 day13_part1_test_13::run_test
        PASS [   0.009s] day13::bin/part2 day13_part2_test_13::run_test
        PASS [   0.011s] day13::bin/part2 day13_part2_13::run_test
        PASS [   0.035s] day12::bin/part1 day12_part1_12::run_test
        PASS [   0.007s] day14::bin/part1 day14_part1_test_14::run_test
        PASS [   0.007s] day14::bin/part2 day14_part2_test_14::run_test
        PASS [   0.007s] day14::bin/part2-order day14_part2_order_test_14::run_test
        PASS [   0.011s] day14::bin/part2-orig day14_part2_orig_test_14::run_test
        PASS [   0.012s] day15::bin/part1 day15_part1_15::run_test
        PASS [   0.024s] day14::bin/part2-v2 day14_part2_v2_test_14::run_test
        PASS [   0.009s] day15::bin/part1 day15_part1_test_15::run_test
        PASS [   0.011s] day15::bin/part2 day15_part2_15::run_test
        PASS [   0.009s] day15::bin/part2 day15_part2_test_15::run_test
        PASS [   0.009s] day16::bin/part1 day16_part1_16::run_test
        PASS [   0.010s] day16::bin/part1 day16_part1_test_16::run_test
        PASS [   0.062s] day14::bin/part1 day14_part1_14::run_test
        PASS [   0.007s] day16::bin/part2 day16_part2_test_16::run_test
        PASS [   0.007s] day16::bin/part2 part1::day16_part1_16::run_test
        PASS [   0.021s] day16::bin/part2 part1::day16_part1_test_16::run_test
        PASS [   0.032s] day17::bin/part1 day17_part1_test_17::run_test
        PASS [   0.075s] day17::bin/part1 day17_part1_17::run_test
        PASS [   0.206s] day10::bin/part2 day10_part2_10::run_test
        PASS [   0.212s] day10::bin/part1 day10_part1_10::run_test
        PASS [   0.034s] day17::bin/part2 day17_part2_test_17::run_test
        PASS [   0.013s] day17::bin/part2 day17_part2_test_17b::run_test
        PASS [   0.020s] day18::bin/part1 day18_part1_18::run_test
        PASS [   0.019s] day18::bin/part1 day18_part1_test_18::run_test
        PASS [   0.141s] day16::bin/part2 day16_part2_16::run_test
        PASS [   0.023s] day18::bin/part1-vertex day18_part1_vertex_18::run_test
        PASS [   0.018s] day18::bin/part1-vertex day18_part1_vertex_test_18::run_test
        PASS [   0.012s] day18::bin/part2 day18_part2_18::run_test
        PASS [   0.014s] day18::bin/part2 day18_part2_test_18::run_test
        PASS [   0.010s] day19::bin/part1 day19_part1_19::run_test
        PASS [   0.011s] day19::bin/part1 day19_part1_test_19::run_test
        PASS [   0.010s] day19::bin/part2 day19_part2_19::run_test
        PASS [   0.008s] day19::bin/part2 day19_part2_test_19::run_test
        PASS [   0.006s] day20::bin/part1 day20_part1_test_20b::run_test
        PASS [   0.008s] day20::bin/part1 day20_part1_test_20::run_test
        PASS [   0.014s] day20::bin/part1 day20_part1_20::run_test
        PASS [   0.009s] day21::bin/part1 day21_part1_21::run_test
        PASS [   0.012s] day21::bin/part1 day21_part1_test_21::run_test
        PASS [   0.007s] day21::bin/part2 day21_part2_test_21::run_test
        PASS [   0.030s] day20::bin/part2 day20_part2_20::run_test
        PASS [   0.012s] day22 types::block_test::test_block_add_point
        PASS [   0.009s] day22 types::block_test::test_block_contains
        PASS [   0.007s] day22 types::block_test::test_block_intersects
        PASS [   0.008s] day22 types::point_test::test_manhattan_distance
        PASS [   0.011s] day22 types::point_test::test_point
        PASS [   0.009s] day22::bin/part1 day22_part1_test_22::run_test
        PASS [   0.010s] day22::bin/part1-settle day22_part1_settle_test_22::run_test
        PASS [   0.037s] day22::bin/part1 day22_part1_22::run_test
        PASS [   0.183s] day17::bin/part2 day17_part2_17::run_test
        PASS [   0.011s] day22::bin/part2 day22_part2_test_22::run_test
        PASS [   0.014s] day22::bin/part2-hash-set day22_part2_hash_set_test_22::run_test
        PASS [   0.718s] day12::bin/part2 day12_part2_12::run_test
        PASS [   0.011s] day22::bin/part2-hash-vec day22_part2_hash_vec_test_22::run_test
        PASS [   0.713s] day22::bin/part2 day22_part2_22::run_test
        PASS [   0.011s] day22::bin/part2-vec-vec day22_part2_vec_vec_test_22::run_test
        PASS [   1.037s] day12::bin/part2-typed day12_part2_typed_12::run_test
        PASS [   0.015s] day23::bin/part1 day23_part1_petgraph_test_23::run_test
        PASS [   0.836s] day21::bin/part2 day21_part2_21::run_test
        PASS [   0.007s] day23::bin/part1-original day23_part1_original_test_23::run_test
        PASS [   0.085s] day23::bin/part1 day23_part1_petgraph_23::run_test
        PASS [   0.011s] day23::bin/part1-petgraph day23_part1_petgraph_test_23::run_test
        PASS [   0.084s] day23::bin/part1-petgraph day23_part1_petgraph_23::run_test
        PASS [   0.011s] day23::bin/part2 day23_part2_petgraph_test_23::run_test
        PASS [   0.010s] day23::bin/part2-brute day23_part2_brute_test_23::run_test
        PASS [   0.012s] day23::bin/part2-path day23_part2_path_test_23::run_test
        PASS [   0.010s] day23::bin/part2-path path_test::test_branching_path
        PASS [   0.015s] day23::bin/part2-path path_test::test_create_path
        PASS [   0.008s] day23::bin/part2-path path_test::test_longer_path
        PASS [   0.421s] day23::bin/part1-original day23_part1_original_23::run_test
        PASS [   0.012s] day23::bin/part2-petgraph day23_part2_petgraph_test_23::run_test
        PASS [   0.015s] day23::bin/part2-splits day23_part2_splits_test_23::run_test
        PASS [   0.012s] day24::bin/part1 day24_part1_24::run_test
        PASS [   1.545s] day12::bin/part1-orig day12_part1_orig_12::run_test
        PASS [   0.012s] day24::bin/part2-brute day24_part2_brute_test_24::run_test
        PASS [   1.575s] day12::bin/part2-brute day12_part2_brute_test_12::run_test
        PASS [   1.571s] day22::bin/part1-settle day22_part1_settle_22::run_test
        PASS [   0.025s] day25::bin/part1 day25_part1_test_25::run_test
        PASS [   0.016s] day25::bin/part1-brute day25_part1_brute_test_25::run_test
        PASS [   2.177s] day14::bin/part2-order day14_part2_order_14::run_test
        PASS [   0.669s] day25::bin/part1 day25_part1_25::run_test
        PASS [   2.228s] day14::bin/part2 day14_part2_14::run_test
        PASS [   2.324s] day14::bin/part2-v2 day14_part2_v2_14::run_test
        PASS [   2.180s] day22::bin/part2-hash-set day22_part2_hash_set_22::run_test
        PASS [   2.257s] day22::bin/part2-hash-vec day22_part2_hash_vec_22::run_test
        PASS [   2.107s] day22::bin/part2-vec-vec day22_part2_vec_vec_22::run_test
        PASS [   1.958s] day23::bin/part2 day23_part2_petgraph_23::run_test
        PASS [   1.899s] day23::bin/part2-petgraph day23_part2_petgraph_23::run_test
        PASS [   2.313s] day24::bin/part2 day24_part2_24::run_test
        PASS [   2.324s] day24::bin/part2-z3 day24_part2_24::run_test
     Summary [   3.939s] 182 tests run: 182 passed, 0 skipped


And then come to think of it… this would actually be a perfect place to put some timing in as well!

fn run_test() {
    let filename = format!("../../data/{}", $input_file);
    let input = std::fs::read_to_string(filename).unwrap();

    let start = std::time::Instant::now();
    let actual = process(input.as_str()).unwrap();
    let elapsed = start.elapsed();

    if std::env::var("AOC_TIMING").is_ok() {
        let output = format!(

        let mut file = OpenOptions::new()
        writeln!(file, "{}", output).unwrap();

    assert_eq!(actual, $expected);

That will write out a single csv with all of the test output, with the $name, $expected and actual value, and timing (both human readable and straight nanoseconds).

A bit of formatting–including dropping all of the test cases and alternate versions and adding line counts–and we have:

daypart 1part
AoC 2023 Day 1: Calibrationinator120.37 µs186.54 µs3553
AoC 2023 Day 2: Playinator65.75 µs59.79 µs391433025
AoC 2023 Day 3: Gearinator582.87 µs477.91 µs1322640
AoC 2023 Day 4: Scratchinator130.70 µs643.79 µs16553350
AoC 2023 Day 5: Growinator47.54 µs150.83 µs192693350
AoC 2023 Day 6: Racinator12.95 µs14.08 µs44302534
AoC 2023 Day 7: Pokinator3.65 ms3.98 ms125262829
AoC 2023 Day 8: Mazinator446.08 µs24.92 ms21473882
AoC 2023 Day 9: Stackinator304.66 µs316.41 µs32163739
AoC 2023 Day 10: Pipinator155.47 ms167.37 ms18930136
AoC 2023 Day 11: Big Banginator422.79 µs424.70 µs1133337
AoC 2023 Day 12: Question Markinator24.64 ms571.92 ms933062183
AoC 2023 Day 13: Reflectinator345.16 µs2.92 ms19751102
AoC 2023 Day 14: Spininator41.19 ms1.68 s1765387
AoC 2023 Day 15: Hashinator53.12 µs190.70 µs2656
AoC 2023 Day 16: Reflectinator513.62 µs93.46 ms289249
AoC 2023 Day 17: A-Starinator28.06 ms108.48 ms467196
AoC 2023 Day 18: Flood Fillinator1.97 ms120.37 µs40625870
AoC 2023 Day 19: Assembly Lininator189.37 µs267.75 µs5411161218
AoC 2023 Day 20: Flip-Flopinator1.65 ms10.16 ms227993135
AoC 2023 Day 21: Step Step Stepinator3.55 ms635.59 ms1467105
AoC 2023 Day 22: Block Dropinator24.57 ms518.85 ms12531102146
AoC 2023 Day 23: Looong Mazinator58.25 ms1.58 s26105194
AoC 2023 Day 24: Collisionator1.33 ms2.25 s263510399
AoC 2023 Day 25: Graph Splitinator579.73 ms4680

My unwritten rule is to write roughly ~100 lines (although I’m far more flexible on this one) and spend less than 1 second on each problem. I only had 3 part 2s over 1 second and even those weren’t bad.

I’m honestly not sure how to improve collisionator if that’s even the time with the Z3 theorem prover…. I suppose optimize what constraints I’m giving it? Sometimes solutions just take time!

Linewise, I’m fairly happy. Lines of code are a fairly terrible metric, but it’s at least a very vague signal? I did get over the mark (more than) a few times. My longest (counting everything) was 383 lines for day 19, part 2.

Actually, only 9 parts are under 100 lines (including imports, whitespace, and comments). But you know… I’m okay with that to!


Overall, it was an interesting year. I miss the programming problem we’ve had in previous years (like 2016's Assembunny and 2015’s days day 7 and day 23).

I’m not a huge fan of problems that reliy on cycle detection (a big favorite this year: day 8, day 14, day 20, day 21), especially when it’s only guaranteed to work based on problem input.

I’m a bit sad that we only got a few animations in this year:

AoC 2023 Day 14: Spininator

AoC 2023 Day 16: Reflectinator

My overall favorites this year:

Another fun year and I think I’m getting better at Rust! If you think I’m totally making that up / have any useful advice, I’d love to hear it though.

And now, time to get back to everything I drop in December to work on these problems. 😄