Have the Advent of Code challenges been getting harder through the month? Are weekend problems more challenging than weekdays? I thought I'd have a look in more detail...

Note: This article was written midway through the 2016 challenge. For updates following the release of all puzzles, see the bottom of this page.

How do you measure coding difficulty?

Was decrypting an MD5 password on December 5th harder than finding out the bathroom keypad on December 2nd? To some extent relative difficulty will depend on personal experience, language of choice and the degree to which you recognise familiar patterns in a problem. But examining the time taken for the fastest 100 people for each day of the AoC gives us some indication of difficulty in a larger (albeit unrepresentative) group. While this might act as some kind of proxy for effort required to solve a problem, it is also worth remembering that those who work through the problems day by day will tend to get better at solving them as they gain experience and re-use previous approaches. The same problem when presented in Day 20 may be easier to solve than if it was presented on Day 2. 

Bearing those observations in mind we can nevertheless say something about how the difficulty of the AoC has progressed through the month. Here is the average time taken for each of the challenges in 2015, showing both the time taken to complete both parts one and two of every two-part challenge.

Day 1 was a bit of an outlier being the first of a new challenge that didn't attract the race to the leaderboard of subsequent days. There is a surprising consistency in time taken throughout the challenge with some obvious anomalies on Day 7 (circuit wiring), Day 19 (reindeer molecules) and Day 22 ( wizard RPG simulator). There was a slight increase in difficulty in the final week with a respite for the final gold start of the challenge on Christmas day. There is an interesting contrast between the two-part structure of Days 19 and 22 with the former having a relatively easy first part, but a second part requiring some reengineering to solve the more complex variant in practical time. This is a pattern that also emerged in the most difficult challenge of the first three weeks of 2016. Day 22 was quite different with most effort required in the initial conceptualisation of the problem.

More generally this division of challenges between those that require significant reengineering for the second more complex part and those that do not is evident in many of the challenges both in 2015 and 2016:

The obvious outlier in 2016 (as of time of writing) was Day 11's lift shifting problem. This was the first (and so far only) problem of 2016 where a naive brute force approach would not yield an answer in any practical time. This therefore required, rather than just encouraged, thought to optimise a solution (by significant pruning of a cyclic search space graph).

It is notable that some of the tougher problems in 2015 and 2016 have occurred at weekends. So are they generally harder than weekdays? The boundary between days is complicated by an international challenge across time zones – the east coast US midnight release of each challenge means that for the leaderboard entrants effectively complete the challenge a the end of the previous day while Europeans and those further east, early the following morning. Bearing this in mind, combining both years' challenge times and viewing by day of week gives some insight:

So it would appear that indeed tougher challenges are given at weekends and that they are more likely to contain the type of puzzle requiring optimised reengineering to solve the second part. It is perhaps just as well that during the week, challenge difficulty eases off towards Friday in preparation for a tough weekend.

Updates

The article above was written on Day 17 (Saturday) of the 2016 challenge. Here are some updates as new challenges are released.

The Saturday night /  Sunday morning Day 18 challenge was easier than had been typical for a weekend in week three so the weekday-weekend difficulty levels have balanced a little.

Sunday night / Monday morning's present steeling problem was uncharacteristically challenging for a Monday particularly as many of the heuristics used for tackling the first part did not apply in the second. This required many to spend time thinking of alternative approaches.

The Monday night / Tuesday morning Day 20 challenge was relatively straightforward and more typical of a mid-week problem, although judging by completion times of the two parts, many on the leaderboard had to re-engineer their solutions somewhat for part two.

Tuesday/Wednesday's challenge was one that took a little time to set up, but was comparatively easy to solve (at least for the first part). This is perhaps an example of a challenge where the time taken by those on the leaderboard is not necessarily indicative of difficulty.

Wednesday/Thursday's grid cell swapping puzzle was one of the harder ones for those who didn't spot the 'trick' subtly hidden in the way the question was expressed. Perhaps those rushing for a leaderboard position are less likely to spot these tricks than if not under time pressure. The final week of the AoC seems to be showing a pattern of more difficult challenges. So far, the Friday questions have tended to be the easier ones. Will the last Friday buck the trend?

The Thursday/Friday assembunny revisited challenge keeps the trend of slightly more time consuming final week. What will the final Christmas weekend bring us?

The penultimate puzzle – robot maze traversal – challenged a few despite having much in common with previous graph traversal problems this year.  It is notable that most of the effort was in constructing a reasonably efficient way of evaluating alternative traversals even to answer the first part. The second part become a trivial addition and is reflected in in the low part two completion time. Overall this brought Saturday up as the most challenging day on average and confirms a final week of tougher puzzles.

As last year, the final Christmas assembunny code challenge didn't tax too many people too much and allowed previous assembunny solutions to be built upon to earn a gold star. This rounded off a slightly more challenging but achievable final Advent of Code week.