Certification Exams: From the Other Side

Today I took the 70-464 beta exam today "Developing Microsoft SQL Server 2012 Databases" and after spending the last few months as an item writer for two other SQL 2012 exams, my perspective on the experience is different.  Having taken numerous SQL 2000, 2005 and 2008 exams, this was my first exam with the benefit of my experience as an item writer.

I obviously can't talk about specifics regarding the exams, and I'll say even less than I'm allowed.

The following are lessons I learned from the experience of being on the item writing team for a certification exam.  I believe these lessons can apply to any technology, for any level of certification exam.

• The wrong answers are more difficult to perfect than the correct answer.  
• Creating challenging wrong answers sometimes involves introducing new variables or twists to your answer, and it is difficult to keep the answer simple.  (More on that later.)
• The right answer is easy, and the polar opposite answer is easy.  But two other wrong answer vectors really is more complicated than you think.  
• We can't make stuff up.  Every answer you see is something real.   The correct answer has to be 100% correct.  The "distractors" have to be plausible. 
• Love your grammar nazi.  
• The reviewer you work with knows nothing about SQL, but everything about proper word choice, Microsoft's preferred phrasing, acceptable acronyms and abbreviations and shorthands.
• It can be pretty frustrating the first time your question gets ripped to shreds because of word issues - not technical issues. Learn from it and try to understand why.  I think I've finally gotten the hang of all the gotchas... just in time to be done.
• You "need to..." not "You must" 
• You "should use" not "could use" or "would use"
• "By using" not "using"
• "Named" not "called"
• "You are developing" not "you are working on"
• "by using the least amount of administrative effort" not "most efficiently" or "best"
• "server that runs SQL Server" not "running SQL"
• If the word doesn't translate well, don't use it.
• The word "only" is a pain in the ass.  
• The "application only connects to one database" vs. the "application connects to only one database" is a big difference.  Think about it.
• "Best practices" are a dangerous minefield.  Just because you as the item writer have always done something one way, doesn't mean it is the only correct way.
• A question cannot rely on "industry standard" practices, but correct and incorrect processes.
• "Least administrative effort" questions need to be airtight - there can be no room for argument.
• When it comes to writing a question, keep it simple.
• A good question tests your grasp of one or two critical pieces of product knowledge.
• A bad question tries to pull in too many different variables or factors and ends up with awkward wording, overly verbose answer text or a non-realistic question.  
• I commented today on a question that obviously grew out of control during the writing process and ended up being unrealistic, with all of the answers undesirable options in an obscure scenario.
• You're given assignments in certain categories of the Objective Domain to write in.  Questions take on a life of their own when you're writing.  Gotta keep on topic, and as a question evolves it is easy to end up with a question in the wrong Objective Domain category.
• Sometimes as the item writer you'll end up writing one question and finishing up another.  Veer off your assignment and you'll find yourself bargaining with the other writers to swap. 
• I only did this once throughout the entire process, and the other writer was understanding and polite.  He let me swap one of my questions that had inadvertently drifted off topic. I took on one of his in a different category, and saved a good question and a lot of effort.  
• It is stressed as an item writer to make sure your question is bound technologically to the Objective Domain category you're writing for.
• Build lists are probably not worth the trouble.
• I developed a couple build lists but they quickly become complicated.  I think they turned out okay, but they were time consuming to nail down a finite, concrete order.
• Build Lists must have one right order.  It is too easy to put unrelated things in the build list that are not dependent.  Build lists give the most problems at alpha. 
• Sure enough, on today's beta exam, I'm 99% sure that one of the Build List questions was broken, by an unlinked chain of events... something that made logical sense but wasn't actually dependent.
• The comments on beta exams are not always... helpful.
• Reviewing beta comments on some of the SQL 2008 exams while on a SME engagement in Redmond was humorous.  Rudely insisting that a question is flawed when you missed the critical bit of info is pretty embarrassing for you on the non-anonymous comment report.
• Oftentimes a question would draw a string of "Answer choices B and C are identical!!!!!!!!1111!!!!!" when there was in fact a significant but tough-to-notice difference. 
• I kept that in mind today when commenting on problems in beta questions. I found a number of typos and incorrect lines, but I withheld the snark and the sarcasm.  No sense in being a jerk about it and making a fool of yourself.
• SME is not "I'm the best person my boss knows" or "I'm pretty good at this".  SME is "I'm probably not going to embarrass myself in front of another SME."