Woven Scoring System and Philosophy

How does Woven score candidates?

By now you've likely seen some Woven scores. What do those scores mean and how do we arrive there?

Same work, same score

Our true north when it comes to scores is that the same work should get the same score. So we're focusing on validity and consistency. We're even willing to make trade offs against precision if we need to.

So that means a 75 is not always strictly better than a 70. On average that is the case, but perhaps not every single time. But we are going to say that 75 if it was submitted three months later, was scored by someone different, and was submitted by someone with a very different resume, it would still get a 75.

Same work. Same score. That's what we hold it to.

Double-blind scoring on a detailed rubric

Now, how do we do that? We start with two scorers who are scoring blind to anything about the candidate and blind to each other. These are engineers who are trained and have to be certified on that scenario in that language before scoring.

They're scoring independently to cancel any bias or error. We also have them scoring to a very detailed rubric. A given scenario will have 20 or more rubric items.

The rubric items are a distillation of expertise down to very objective criteria. So they might be as complex as "did they solve this problem in a functional manner?" or "did they indicate the missing index on the database will be a likely cause of this specific outage?" down to something more simple like, "did they remember to complement any part of this pull request when they're doing a code review for a colleague?"

They're scoring each one of those rubric items: did they meet the criteria or not? Those rubric items are summed up and weighted, some of them are more or less important, and they create a final score. Even if the two scorers agree on the overall score, if they disagree on any single rubric item, we have a third engineer who's more senior go in and QA that so that you can really trust that even at a rubric item level, it's correct.

That means the same work gets the same score.

Doesn't this take a long time?

Now that's a lot. You might be thinking, well, I don't have time for all that. Because senior engineers are not in the market for long. We've gotta move fast.

We've been doing this a long time. You can expedite a candidate, and it's typically turned around within two hours or even faster.

And if you just let the system run, over 99% of results are returned same or next business day.

You can be confident in moving fast. Because the score is giving you consistent information and you're getting it fast enough to win those experience candidates.