You may have seen Shaan Hurley and I having a discussion (ahem) over the validity of his statement:
I really do use the ribbon now with AutoCAD 2010 along with most users as evidenced by the CIP data we receive daily from thousands of AutoCAD users who choose to send the great data.
So, now you know. Most of you use the Ribbon now, Shaan said so. Shaan, as he always has done in the past, declined my invitation to back up this assertion with more details. He has vast amounts of data collected from huge numbers of users. How could that possibly be wrong?
Here’s how. CIP data is biased.
How can millions of data points be biased? Actually, all samples are biased. Only the degree of bias varies. The polls on this blog are no exception. I do my best to keep the questions and options neutral; the only leading questions you’ll see here in serious polls are the ones I copy and paste from Autodesk blogs. But readers of this blog are one self-selecting small portion of Autodesk customers, and people who vote in my polls represent another self-selecting portion of that portion.
The question is, how biased is Autodesk’s CIP data? Without access to Autodesk’s data (which it won’t provide) and resources for alternative data collection from its customers (ditto), the best I can do is use my own biased sample (that’s you lot out there) as a cross-check.
Let’s examine it in light of Ribbon use among AutoCAD 2010 users. In an earlier comparison of my 2009 poll figures and Shaan’s CIP data, I wrote this:
But Shaan’s CIP users are also a biased sample, comprising those AutoCAD users who have CIP turned on. Are users who go with the flow and have CIP on also more likely to go with the flow and leave the Ribbon on? Possibly, but I would have thought the CIP-on bias would be less significant than the blog-reader bias.
I have recently run a poll to try to determine if that “possibly”, that hunch, has any basis. Let’s examine the results I got.
AutoCAD 2010 users, what are your Ribbon and CIP settings?
Ribbon on, CIP on (24.7%, 65 Votes)
Ribbon on, CIP off (19.4%, 51 Votes)
Ribbon off, CIP on (11%, 29 Votes)
Ribbon off, CIP off (44.9%, 118 Votes)
Total Voters: 263
For the sake of argument, let’s make the assumption that my poll sample is unbiased. It’s not, and the degree of bias is unknown, but let’s see what it would mean if it was. Let’s see what kind of results Autodesk would see from its CIP sample:
CIP-on voters (94):
Ribbon on 69% (65)
Ribbon off 31% (29)
Shaan would see from this result that 69% of AutoCAD 2010 users have the Ribbon on, and would be tempted to say stuff like “use the ribbon now with AutoCAD 2010 along with most users”. Understandable. That’s just CIP users, but non-CIP users can’t be that different, surely? Or can they?
CIP-off voters (169):
Ribbon on 30% (51)
Ribbon off 70% (118)
Wow. That’s a huge discrepancy, and it implies that a sample selection based on CIP use introduces a massive bias. I’ve watched this poll grow over the weeks, half-expecting things to even out as the sample size increased. It didn’t. It has been pretty constant, with non-Ribbon non-CIP users outnumbering Ribbonite non-CIP users by a substantial margin.
Let’s put the groups together, shall we?
All voters (263)
Ribbon on 44% (116)
Ribbon off 56% (147)
So, if the voters in my poll were observed by Autodesk via CIP they would appear to be 69% Ribbon users. In fact, only 44% of these voters are Ribbon users.
How many AutoCAD 2010 users really have the Ribbon on? 69%? 30%? 44%? Some other number? I don’t know, and that’s not the point. The point is, Autodesk doesn’t know either. It can take some smart guesses, but just assuming CIP is accurate isn’t smart, it’s just a guess.
Why does this matter? Because Autodesk makes decisions based on this stuff. Decisions that affect you and me and how we use our tools. Have a look at this statement from Autodesk’s Teresa Anania, Director of Industry Management (taken from her interview with Deelip Menezes about Inventor):
…we had data that suggested that the new ribbon UI was well received and would be absolutely all that customers needed …. And now since we have the CIP data that shows us how our customers are using the software, we can analyze this before we permanently turn anything off.
Comments like this (and others from other Adeskers) seem to indicate that there is an unspoken assumption that CIP users accurately represent a true cross-section of users in general.
I know that Autodesk doesn’t rely solely on CIP; it uses a wide range of research tools to find out what users are up to and what they need. I regularly encourage you to participate in various Autodesk surveys, for example. But there are problems of accuracy inherent in all those methods. It would be natural, when faced with a set of apparently conflicting results from different sources, for Autodesk decision-makers to simply assume that the source with the biggest sample size is the most accurate. That could be a dangerous mistake, for both Autodesk and its customers.
Note: my arithmetic was off in several places when I posted this, and I have edited the post to correct some of the figures. These corrections do not invalidate the arguments; the substantial bias is still evident.
Autodesk also suffers from confirmation bias. They want to believe that users are embracing the ribbon, and so they seek out data that confirms their belief.
Ribbon use cant really be decided based on whether you have the ribbon on or off. Mine is hidden, anchored to the left side of the screen. I may use it once every 10 or 11 sessions of AutoCAD, or even more rarely than that. Even then, I only use it for my custom routines as it replaced my pulldown menu. Also every time I use the block editor, I am forced to use the ribbon, those numbers get skewed as well. If I remember right, for 2011, we decided to shut off the CIP because of the bugs associated with it.
Chris, it’s fair to say that my poll doesn’t adequately allow for partial usage like yours.
Does CIP adequately allow for it? Does it count you as a Ribbon user or not? Dunno. Autodesk declines to say exactly how these things are counted. I’ve asked more than once and no information has been forthcoming. People will have to draw their own conclusions about that.
With 2011 the Ribbon provides a lot more useful tolls (i.e. faster hatching) and the delay seems to be pretty much gone with the Ribbon, at least on our systems here. Of course I have heavily customized the ribbon to put the commands that people don’t normally type or use frequently within easy reach. This default of having things like line be so prevalent is dumb, no one clicks a button for line, why should they when l and space is much faster? I have also modified the hatch editor and mtext editor ribbons to eliminate as many extra clicks as I can. So, once properly customized the Ribbon can be a great tool.
I agree, I really dont think that any data collected, via CIP or a poll can allow for partial usage, especially when they only check for on/off status. I’m not saying that these are incorrect methods of collecting the data. If you, Steve, were to allow for every answer possible, your pole would probably have one vote for each choice and would not really provide any conclusive results. The method you have used, does provide conclusive results (although the sample set is skewed) However, when it comes to the CIP, I would think that it would be possible for Autodesk to look at real usage, how many commands are accessed via the ribbon per AutoCAD session. (although the sample set is skewed as well) However, since they decline to say how things are counted, you are correct, people must draw their own conclusions.
CIP or not, none of the data collected shows the number of users (like myself) who have forsaken the new and improved post-2009 releases and reloaded 2008 to regain productive operation – or refuse to be forced to used the idiotic UI overlay (dark & light) schemes. I’ve gone a step further and have restored the MNU/MNS menu system to avoid the nasty results that the CUI can pump out. No cartoons in the pulldowns, two clicks away from any top-level toolbar command (dynamic toolbar system lifted from MDT), no redundant and annoying big scarlet letter Menu Browser/Application Menu, no redundant Quick Access Toolbar, no (no 2008 updates – no longer valid) InfoCenter toolbar. And a clean simple Windows UI that doesn’t flash on/off to display the underlying standard Windows UI.
Where’s the Autodesk poll seeking the level of annoyance that customers are willing to tolerate? More importantly, where are the previously built-in options which allow users to turn the new and improved crap off? Are a handful of new and/or improved (some remain incomplete and/or unfinished) commands worth the loss in overall productivity? Not to me. Will 2012 fix the problems? That’s only a ten month (non-alpha/beta) wait away.
to put a completely different spin on this thread… how about completely ignoring what Autodesk (or any outside agency) polls show and looking to one’s own organizational behaviors for relevant decision-making? Autodesk’s information may be valid for the markets and userbase they are surveying, but that doesn’t necessarily make their poll feedback relevant to any other organizations’ needs, skill levels, automations, processes, cultures or technological maturities. how do i say this? Autodesk gives our organization no context for their polls, which makes them… amusing, but not helpful. for example, if 100% of Shaan’s polls were all taken by first month AutoCAD users, i would fully expect all of them to love and use the ribbon interface. i imagine the feedback would be very different if these pools were taken by AutoCAD 15-20 year vets (which is typical of our organization’s userbase). here’s what i discovered about our organization and the ribbon: over 90% of our users, having been trained completely on the ribbon interface and instructed to test it out for 2 weeks, immediately disabled it when given the opportunity. when asked why they did so, they said it slowed them down, it took up too much screen realty, it had too many vital commands burried deep in the new hierarchy to make it worthwhile, and certain vital commands they needed to get their work done were not available at all. in short, they saw limited to no advantage in the (yet again) re-organized toolsets they relied on to get their work done. curiously enough, another 5% of our users who had’t disabled the ribbon did so later pn after we discovered it was because they did not know or remember how to disable it. the remaining 5% who loved the ribbon were users with less than 2 years of AutoCAD experience. a “big friendly buttons with big friendly pictures” (aka DON’T PANIC) UI didn’t impress our hardcores who represent the majority of our users. they are typists with highly customized keyboard aliases who either fly at high speed thru the command prompt interface or automate the daylights out of their processes with personal scripts. for them, the ribbon interface was like going from DSL to dial-up. having used the UI myself, i completely understand their reasoning and their choices. our organization’s feedback is not definitive of anyone else’s, nor does it invalidate Autodesk’s own. our feedback is simply… 100% relevant.
Dave’s post pretty much hit my thoughts, so I won’t repeat them. Does it seem to anyone else that each year, questions like “how do I disable feature ‘X'”, become more and more prevalent? Are we paying for ‘features’ we’ll never use?
I really didn’t intend this post to be about the Ribbon as such, but if that’s what you want to discuss, that’s OK.
R.K., to be fair, this is not new. We have always been paying for features we don’t use. Autodesk can’t please everybody with everything it adds to its products. We have to be content with having at least some useful additions each time, no removal of useful features, plus the ability to turn off or ignore any useless new stuff.
Remember AutoCAD 2000i, quite possibly the definitive weak release? People remember it for its its intrusive interface (Active Hindrance!) and the removal of Express Tools in an ill-advised cash grab attempt. That feeble effort of a release contained some useful stuff that most users wouldn’t want to do without these days (double-click editing is the standout), but there is no way it represented a good value for money upgrade. And that’s a decade ago now.
re: Does it count you as a Ribbon user or not?
I haven’t seen Autodesk CIP data but Microsoft Office CIP (based on example seen in a ribbon presentation) captured both the command use, flow and the input method. They could distingush between copy (or whatever) via keyboard, menu or toolbar.
I assume, hope, Autodesk are able to do the same so a keyed “L”, toolbar or ribbom initated Line could be reflected in the analysis.