A quick follow up to Advent of Code 2023: testing and timing.
It’s been bothering me a bit that I haven’t had a generic way to run tests and timing on every problem as I’m going.
So let’s fix it!
Testing
I’ve been filling out test data with a theoretical future macro like this:
// #[aoc_test("data/test/01.txt", 142)]
// #[aoc_test("data/test/01b.txt", 209)]
// #[aoc_test("data/01.txt", 53651)]
Which almost works (and would be fine if we were using a procedural macro). But if we want to use a more simple declarative macro with macro_rules! instead, we have to deal with macro hygiene. Basically, we can’t create new names (which we’d love to do for the module names).
So instead, I ended up with this:
aoc_test::generate!{day01_part1_test_01 as "test/01.txt" => "142"}
aoc_test::generate!{day01_part1_test_01b as "test/01b.txt" => "209"}
aoc_test::generate!{day01_part1_01 as "01.txt" => "53651"}
A bit longer than I’d like, but workable. And with a bit of regex (and a script to actually get the package and binary name), we can automatically rewrite the old form to this.
The actual macro:
#[macro_export]
macro_rules! generate {
($name:tt as $input_file:tt => $expected:tt) => {
#[cfg(test)]
mod $name {
use super::*;
use std::fs::OpenOptions;
use std::io::prelude::*;
#[test]
fn run_test() {
let filename = format!("../../data/{}", $input_file);
let input = std::fs::read_to_string(filename).unwrap();
let actual = process(input.as_str()).unwrap();
assert_eq!(actual, $expected);
}
}
};
}
That’s really it! Running it with cargo test we get a giant pile of output. cargo nextest does much better:
$ cargo nextest run --release
warning: crate `day__day__` should have a snake case name
|
= help: convert the identifier to snake case: `day_day`
= note: `#[warn(non_snake_case)]` on by default
warning: `day__day__` (lib) generated 1 warning
warning: `day__day__` (lib test) generated 1 warning (1 duplicate)
Finished release [optimized] target(s) in 0.05s
Starting 182 tests across 119 binaries
PASS [ 0.007s] day01::bin/part1 day01_part1_test_01b::run_test
PASS [ 0.005s] day01::bin/part1-first-last day01_part1_first_last_01::run_test
PASS [ 0.009s] day01::bin/part1 day01_part1_test_01::run_test
PASS [ 0.012s] day01::bin/part1 day01_part1_01::run_test
PASS [ 0.005s] day01::bin/part2 day01_part2_01::run_test
PASS [ 0.006s] day01::bin/part1-first-last tests::test_first_and_last
PASS [ 0.006s] day01::bin/part1-first-last day01_part1_first_last_test_01::run_test
PASS [ 0.008s] day01::bin/part2 day01_part2_test_01::run_test
PASS [ 0.007s] day02 parse::tests::test_games
PASS [ 0.007s] day02 parse::tests::test_game
PASS [ 0.008s] day02 parse::tests::test_color
PASS [ 0.008s] day01::bin/part2 day01_part2_test_01b::run_test
PASS [ 0.005s] day02 parse::tests::test_round
PASS [ 0.005s] day02::bin/part1 day02_part1_02::run_test
PASS [ 0.005s] day02::bin/part1 day02_part1_test_02::run_test
PASS [ 0.006s] day02::bin/part2 day02_part2_02::run_test
PASS [ 0.006s] day02::bin/part2 day02_part2_test_02::run_test
PASS [ 0.008s] day03 types::tests::test_neighbor
PASS [ 0.006s] day03 types::tests::test_parse_longer_number
PASS [ 0.007s] day03 types::tests::test_parse_numer
PASS [ 0.007s] day03::bin/part2 day03_part2_03::run_test
PASS [ 0.007s] day03::bin/part2 day03_part2_test_03::run_test
PASS [ 0.008s] day03 types::tests::test_symbols
PASS [ 0.007s] day04 parse::test::test_parse_card_whitepsace
PASS [ 0.007s] day04 parse::test::test_parse_card
PASS [ 0.010s] day03::bin/part1 day03_part1_03::run_test
PASS [ 0.010s] day03::bin/part1 day03_part1_test_03::run_test
PASS [ 0.006s] day04::bin/part2 day04_part2_04::run_test
PASS [ 0.007s] day04::bin/part1 day04_part1_test_04::run_test
PASS [ 0.008s] day04::bin/part1 day04_part1_04::run_test
PASS [ 0.008s] day04::bin/part2 day04_part2_test_04::run_test
PASS [ 0.008s] day05 types::test::test_category_map_apply
PASS [ 0.006s] day05::bin/part1 day05_part1_05::run_test
PASS [ 0.006s] day05::bin/part1 day05_part1_test_05::run_test
PASS [ 0.007s] day05 types::test::test_range_map_apply
PASS [ 0.005s] day05::bin/part2 day05_part2_test_05::run_test
PASS [ 0.008s] day05 types::test::test_range_map_range_apply
PASS [ 0.006s] day05::bin/part2 day05_part2_05::run_test
PASS [ 0.007s] day05::bin/part2-brute day05_part2_brute_test_05::run_test
PASS [ 0.006s] day06::bin/part1 day06_part1_06::run_test
PASS [ 0.006s] day06::bin/part1 day06_part1_test_06::run_test
PASS [ 0.005s] day06::bin/part1-brute day06_part1_brute_test_06::run_test
PASS [ 0.007s] day06::bin/part1-brute day06_part1_brute_06::run_test
PASS [ 0.007s] day06::bin/part2 day06_part2_06::run_test
PASS [ 0.005s] day06::bin/part2-brute day06_part2_brute_test_06::run_test
PASS [ 0.006s] day06::bin/part2 day06_part2_test_06::run_test
PASS [ 0.005s] day07::bin/part2 day07_part2_test_07::run_test
PASS [ 0.013s] day05::bin/part2-brute-par day05_part2_brute_par_test_05::run_test
PASS [ 0.007s] day08::bin/part1 day08_part1_08::run_test
PASS [ 0.009s] day07::bin/part1 day07_part1_test_07::run_test
PASS [ 0.005s] day08::bin/part2 day08_part2_test_08c::run_test
PASS [ 0.011s] day07::bin/part1 day07_part1_07::run_test
PASS [ 0.009s] day08::bin/part1 day08_part1_test_08::run_test
PASS [ 0.008s] day08::bin/part2 day08_part2_test_08::run_test
PASS [ 0.011s] day07::bin/part2 day07_part2_07::run_test
PASS [ 0.007s] day08::bin/part2 day08_part2_test_08b::run_test
PASS [ 0.006s] day08::bin/part2-brute day08_part2_brute_test_08c::run_test
PASS [ 0.008s] day08::bin/part2-brute day08_part2_brute_test_08b::run_test
PASS [ 0.009s] day08::bin/part2-brute day08_part2_brute_test_08::run_test
PASS [ 0.008s] day09::bin/part1 day09_part1_09::run_test
PASS [ 0.006s] day09::bin/part1 day09_part1_test_09::run_test
PASS [ 0.006s] day09::bin/part2 day09_part2_09::run_test
PASS [ 0.006s] day10::bin/part1 day10_part1_test_10b::run_test
PASS [ 0.006s] day09::bin/part2 day09_part2_test_09::run_test
PASS [ 0.007s] day10::bin/part1 day10_part1_test_10::run_test
PASS [ 0.007s] day10::bin/part1 day10_part1_test_10c::run_test
PASS [ 0.007s] day10::bin/part2 day10_part2_test_10b::run_test
PASS [ 0.009s] day10::bin/part2 day10_part2_test_10::run_test
PASS [ 0.007s] day10::bin/part2 day10_part2_test_10e::run_test
PASS [ 0.008s] day10::bin/part2 day10_part2_test_10c::run_test
PASS [ 0.006s] day10::bin/part2 day10_part2_test_10f::run_test
PASS [ 0.027s] day06::bin/part2-brute day06_part2_brute_06::run_test
PASS [ 0.008s] day10::bin/part2 day10_part2_test_10d::run_test
PASS [ 0.008s] day11::bin/part1 day11_part1_11::run_test
PASS [ 0.007s] day11::bin/part1 day11_part1_test_11::run_test
PASS [ 0.006s] day11::bin/part2 day11_part2_test_11::run_test
PASS [ 0.007s] day12::bin/part1 day12_part1_test_12::run_test
PASS [ 0.008s] day11::bin/part2 day11_part2_11::run_test
PASS [ 0.006s] day12::bin/part1-orig day12_part1_orig_test_12::run_test
PASS [ 0.009s] day12::bin/part2 day12_part2_test_12::run_test
PASS [ 0.009s] day12::bin/part2 tests::hash_question
PASS [ 0.006s] day12::bin/part2 tests::simple_questions
PASS [ 0.009s] day12::bin/part2 tests::no_questions
PASS [ 0.039s] day08::bin/part2 day08_part2_08::run_test
PASS [ 0.007s] day13 types::bounds_test::test_contains
PASS [ 0.009s] day13 types::bounds_test::test_include
PASS [ 0.011s] day12::bin/part2-typed day12_part2_typed_test_12::run_test
PASS [ 0.010s] day13 types::point_test::test_reflect_x
PASS [ 0.009s] day13::bin/part1 day13_part1_13::run_test
PASS [ 0.012s] day13 types::point_test::test_reflect_y
PASS [ 0.009s] day13::bin/part1 day13_part1_test_13::run_test
PASS [ 0.009s] day13::bin/part2 day13_part2_test_13::run_test
PASS [ 0.011s] day13::bin/part2 day13_part2_13::run_test
PASS [ 0.035s] day12::bin/part1 day12_part1_12::run_test
PASS [ 0.007s] day14::bin/part1 day14_part1_test_14::run_test
PASS [ 0.007s] day14::bin/part2 day14_part2_test_14::run_test
PASS [ 0.007s] day14::bin/part2-order day14_part2_order_test_14::run_test
PASS [ 0.011s] day14::bin/part2-orig day14_part2_orig_test_14::run_test
PASS [ 0.012s] day15::bin/part1 day15_part1_15::run_test
PASS [ 0.024s] day14::bin/part2-v2 day14_part2_v2_test_14::run_test
PASS [ 0.009s] day15::bin/part1 day15_part1_test_15::run_test
PASS [ 0.011s] day15::bin/part2 day15_part2_15::run_test
PASS [ 0.009s] day15::bin/part2 day15_part2_test_15::run_test
PASS [ 0.009s] day16::bin/part1 day16_part1_16::run_test
PASS [ 0.010s] day16::bin/part1 day16_part1_test_16::run_test
PASS [ 0.062s] day14::bin/part1 day14_part1_14::run_test
PASS [ 0.007s] day16::bin/part2 day16_part2_test_16::run_test
PASS [ 0.007s] day16::bin/part2 part1::day16_part1_16::run_test
PASS [ 0.021s] day16::bin/part2 part1::day16_part1_test_16::run_test
PASS [ 0.032s] day17::bin/part1 day17_part1_test_17::run_test
PASS [ 0.075s] day17::bin/part1 day17_part1_17::run_test
PASS [ 0.206s] day10::bin/part2 day10_part2_10::run_test
PASS [ 0.212s] day10::bin/part1 day10_part1_10::run_test
PASS [ 0.034s] day17::bin/part2 day17_part2_test_17::run_test
PASS [ 0.013s] day17::bin/part2 day17_part2_test_17b::run_test
PASS [ 0.020s] day18::bin/part1 day18_part1_18::run_test
PASS [ 0.019s] day18::bin/part1 day18_part1_test_18::run_test
PASS [ 0.141s] day16::bin/part2 day16_part2_16::run_test
PASS [ 0.023s] day18::bin/part1-vertex day18_part1_vertex_18::run_test
PASS [ 0.018s] day18::bin/part1-vertex day18_part1_vertex_test_18::run_test
PASS [ 0.012s] day18::bin/part2 day18_part2_18::run_test
PASS [ 0.014s] day18::bin/part2 day18_part2_test_18::run_test
PASS [ 0.010s] day19::bin/part1 day19_part1_19::run_test
PASS [ 0.011s] day19::bin/part1 day19_part1_test_19::run_test
PASS [ 0.010s] day19::bin/part2 day19_part2_19::run_test
PASS [ 0.008s] day19::bin/part2 day19_part2_test_19::run_test
PASS [ 0.006s] day20::bin/part1 day20_part1_test_20b::run_test
PASS [ 0.008s] day20::bin/part1 day20_part1_test_20::run_test
PASS [ 0.014s] day20::bin/part1 day20_part1_20::run_test
PASS [ 0.009s] day21::bin/part1 day21_part1_21::run_test
PASS [ 0.012s] day21::bin/part1 day21_part1_test_21::run_test
PASS [ 0.007s] day21::bin/part2 day21_part2_test_21::run_test
PASS [ 0.030s] day20::bin/part2 day20_part2_20::run_test
PASS [ 0.012s] day22 types::block_test::test_block_add_point
PASS [ 0.009s] day22 types::block_test::test_block_contains
PASS [ 0.007s] day22 types::block_test::test_block_intersects
PASS [ 0.008s] day22 types::point_test::test_manhattan_distance
PASS [ 0.011s] day22 types::point_test::test_point
PASS [ 0.009s] day22::bin/part1 day22_part1_test_22::run_test
PASS [ 0.010s] day22::bin/part1-settle day22_part1_settle_test_22::run_test
PASS [ 0.037s] day22::bin/part1 day22_part1_22::run_test
PASS [ 0.183s] day17::bin/part2 day17_part2_17::run_test
PASS [ 0.011s] day22::bin/part2 day22_part2_test_22::run_test
PASS [ 0.014s] day22::bin/part2-hash-set day22_part2_hash_set_test_22::run_test
PASS [ 0.718s] day12::bin/part2 day12_part2_12::run_test
PASS [ 0.011s] day22::bin/part2-hash-vec day22_part2_hash_vec_test_22::run_test
PASS [ 0.713s] day22::bin/part2 day22_part2_22::run_test
PASS [ 0.011s] day22::bin/part2-vec-vec day22_part2_vec_vec_test_22::run_test
PASS [ 1.037s] day12::bin/part2-typed day12_part2_typed_12::run_test
PASS [ 0.015s] day23::bin/part1 day23_part1_petgraph_test_23::run_test
PASS [ 0.836s] day21::bin/part2 day21_part2_21::run_test
PASS [ 0.007s] day23::bin/part1-original day23_part1_original_test_23::run_test
PASS [ 0.085s] day23::bin/part1 day23_part1_petgraph_23::run_test
PASS [ 0.011s] day23::bin/part1-petgraph day23_part1_petgraph_test_23::run_test
PASS [ 0.084s] day23::bin/part1-petgraph day23_part1_petgraph_23::run_test
PASS [ 0.011s] day23::bin/part2 day23_part2_petgraph_test_23::run_test
PASS [ 0.010s] day23::bin/part2-brute day23_part2_brute_test_23::run_test
PASS [ 0.012s] day23::bin/part2-path day23_part2_path_test_23::run_test
PASS [ 0.010s] day23::bin/part2-path path_test::test_branching_path
PASS [ 0.015s] day23::bin/part2-path path_test::test_create_path
PASS [ 0.008s] day23::bin/part2-path path_test::test_longer_path
PASS [ 0.421s] day23::bin/part1-original day23_part1_original_23::run_test
PASS [ 0.012s] day23::bin/part2-petgraph day23_part2_petgraph_test_23::run_test
PASS [ 0.015s] day23::bin/part2-splits day23_part2_splits_test_23::run_test
PASS [ 0.012s] day24::bin/part1 day24_part1_24::run_test
PASS [ 1.545s] day12::bin/part1-orig day12_part1_orig_12::run_test
PASS [ 0.012s] day24::bin/part2-brute day24_part2_brute_test_24::run_test
PASS [ 1.575s] day12::bin/part2-brute day12_part2_brute_test_12::run_test
PASS [ 1.571s] day22::bin/part1-settle day22_part1_settle_22::run_test
PASS [ 0.025s] day25::bin/part1 day25_part1_test_25::run_test
PASS [ 0.016s] day25::bin/part1-brute day25_part1_brute_test_25::run_test
PASS [ 2.177s] day14::bin/part2-order day14_part2_order_14::run_test
PASS [ 0.669s] day25::bin/part1 day25_part1_25::run_test
PASS [ 2.228s] day14::bin/part2 day14_part2_14::run_test
PASS [ 2.324s] day14::bin/part2-v2 day14_part2_v2_14::run_test
PASS [ 2.180s] day22::bin/part2-hash-set day22_part2_hash_set_22::run_test
PASS [ 2.257s] day22::bin/part2-hash-vec day22_part2_hash_vec_22::run_test
PASS [ 2.107s] day22::bin/part2-vec-vec day22_part2_vec_vec_22::run_test
PASS [ 1.958s] day23::bin/part2 day23_part2_petgraph_23::run_test
PASS [ 1.899s] day23::bin/part2-petgraph day23_part2_petgraph_23::run_test
PASS [ 2.313s] day24::bin/part2 day24_part2_24::run_test
PASS [ 2.324s] day24::bin/part2-z3 day24_part2_24::run_test
------------
Summary [ 3.939s] 182 tests run: 182 passed, 0 skipped
Timing
And then come to think of it… this would actually be a perfect place to put some timing in as well!
#[test]
fn run_test() {
let filename = format!("../../data/{}", $input_file);
let input = std::fs::read_to_string(filename).unwrap();
let start = std::time::Instant::now();
let actual = process(input.as_str()).unwrap();
let elapsed = start.elapsed();
if std::env::var("AOC_TIMING").is_ok() {
let output = format!(
"{},{},{},{:?},{}",
stringify!($name),
$expected,
actual,
elapsed,
elapsed.as_nanos()
);
let mut file = OpenOptions::new()
.create(true)
.append(true)
.open("../timing.csv")
.unwrap();
writeln!(file, "{}", output).unwrap();
}
assert_eq!(actual, $expected);
}
That will write out a single csv with all of the test output, with the $name, $expected and actual value, and timing (both human readable and straight nanoseconds).
A bit of formatting–including dropping all of the test cases and alternate versions and adding line counts–and we have:
My unwritten rule is to write roughly ~100 lines (although I’m far more flexible on this one) and spend less than 1 second on each problem. I only had 3 part 2s over 1 second and even those weren’t bad.
I’m honestly not sure how to improve collisionator if that’s even the time with the Z3 theorem prover…. I suppose optimize what constraints I’m giving it? Sometimes solutions just take time!
Linewise, I’m fairly happy. Lines of code are a fairly terrible metric, but it’s at least a very vague signal? I did get over the mark (more than) a few times. My longest (counting everything) was 383 lines for day 19, part 2.
Actually, only 9 parts are under 100 lines (including imports, whitespace, and comments). But you know… I’m okay with that to!
Summary
Overall, it was an interesting year. I miss the programming problem we’ve had in previous years (like 2016's Assembunny and 2015’s days day 7 and day 23).
I’m not a huge fan of problems that reliy on cycle detection (a big favorite this year: day 8, day 14, day 20, day 21), especially when it’s only guaranteed to work based on problem input.
I’m a bit sad that we only got a few animations in this year:
AoC 2023 Day 16: Reflectinator
My overall favorites this year:
- AoC 2023 Day 7: Pokinator - Implementing a custom
Ordwas neat. - AoC 2023 Day 12: Question Markinator - An example of how powerful caching/dynamic programming can be.
- AoC 2023 Day 16: Reflectinator - Lasers! And an animation.
- AoC 2023 Day 19: Assembly Lininator - I guess this does count as a virtual machine? I actually like the ‘apply to a range’ problems.
- AoC 2023 Day 20: Flip-Flopinator - Circuit simulations are pretty cool and almost a virtual machine!
- AoC 2023 Day 24: Collisionator - I actually enjoyed the brute force solution to that one (after trying with the solver).
Another fun year and I think I’m getting better at Rust! If you think I’m totally making that up / have any useful advice, I’d love to hear it though.
And now, time to get back to everything I drop in December to work on these problems. 😄
Onward!