Tag Archives: Productivity

Studying Autodesk’s productivity study

8 Replies

Heidi Hewett just reported the following on her blog, about a productivity study:

According to a recent independent study, AutoCAD® 2011 can help you work up to 44% faster with the latest productivity enhancements.

I have a couple of problems with that sentence. First, it’s not an independent study. It’s a study conducted by long-time respected CAD figure David Cohn, but it was specified and paid for by Autodesk:

This productivity study was performed at the request of Autodesk Inc., which funded this work.

That’s not exactly independent then, is it? Second, the study does not state that AutoCAD 2011 is responsible for a 44% improvement. That’s a figure that combines both the effects of AutoCAD 2011 (over AutoCAD 2008), plus the effects of using a newer, faster PC. Just stating that figure wthout such a disclaimer is misleading.

Now to the study itself. Let me make it clear that I have no problem with David Cohn, who is respected, experienced and honest. I do not doubt that his study accurately describes his observations of the time taken to perform the chosen operations on the chosen drawings. The problem is that the study is designed to concentrate purely on a set of AutoCAD operations that benefit from the changes of the last three releases. In other words, the dice are very heavily loaded. To David’s credit, he states that very clearly in the study report:

Each drawing was chosen based on a number of criteria designed to showcase one or more features of the software that did not exist in AutoCAD 2008 but were added in subsequent releases. While each drawing could certainly be produced using the features and functions available in AutoCAD 2008, the advanced capabilities added in subsequent releases would likely enable a typical user to produce the drawing faster using AutoCAD 2011.

Since the premise of the test was to determine how much time could be saved by using a new feature, the test itself was already predisposed to show that using AutoCAD 2011 is more productive than using AutoCAD 2008.

A quick skim-read shows that there are several other problems with the study. For example, it doesn’t attempt to measure the productivity of those operations that are common to both releases, which are much more likely to be used in bulk by typical users. The report states that the Ribbon interface is likely to be more productive, but makes no attempt to justify that by comparing the exact same operations performed using the two interfaces.

In addition, both AutoCAD 2008 and 2011 are measured on a typical middle-age PC using XP, but only 2011 is measured on a modern PC running Windows 7. The report states that the latter tests were performed after the former tests, so the times will also be biased by familiarity with AutoCAD 2011, the drawings and the operations required. That’s where the 44% figure comes from, and it doesn’t mean anything.

What’s the point of studies like this, that are self-evidently designed to produce a good-looking outcome? Who are they supposed to fool?  Come on Autodesk, either do these things properly or don’t do them at all. Please.

AutoCAD performance and productivity

8 Replies

I have closed the performance and productivity polls as described in my posts here and here, and the results can be seen in the Polls Archive. As with most of the other polls I’ve run here, the distribution of votes has not changed greatly after the first few days.

It is clear from the very different voting patterns in the two polls that blog nauseam readers are smart enough understand the difference between the two questions. The performance poll has a very clear skew to the “slower” side. This supports the empirical evidence I’ve seen elsewhere that people perceive AutoCAD as getting slower. This is stuff they’ve noticed for themselves, not a few milliseconds here and there.

On the other hand, the productivity poll results show a much more even distribution. The five options are pretty equally represented, except that “a lot more productive” has suffered at the hands of the most popular choice, “a bit more productive”. If you calculate the mean result, it is almost bang in the middle. It’s actually slightly worse than that, but by such a very small margin that it is not statistically significant.

Overall, we can say that the average viewpoint expressed here is that people clearly see AutoCAD as getting slower, but that its productivity has stayed about the same. So, does this let Autodesk off the performance hook? If a slowing AutoCAD is balanced by productivity-enhancing features, does performance matter? In my opinion, the answers to those questions are no and yes respectively.

It’s not a safe assumption that productivity features are balancing speed issues for everybody across the board. A new feature may help some users’ productivity, maybe even a majority of users, but it won’t help everybody. Some new features even harm some people’s productivity, which is one more reason for being grateful that Autodesk generally lets us turn them off (although it has been forgetful about that in some cases in recent years). Performance is one of the many things that impact productivity, but unlike most new features, is something that impacts the productivity of everybody. Even a first-time user has to sit around waiting for AutoCAD to start up, while a fast power user will be rendered less productive, and certainly more frustrated, by relatively small hesitations.

Furthermore, if AutoCAD is about as productive as it was a few releases back, is that good enough? Or should Autodesk be providing noticeably more productivity in return for our Subscription or upgrade payments? If not, why do we continue to hand over our hard-earned dollars? Why do we go through the upgrade process at all, with all its attendant costs, struggles and inconveniences?

Autodesk, please put much more effort into halting and reversing AutoCAD’s performance slide. It doesn’t have to be a competition between performance and productivity. Improve the former and the latter will also improve.

Is AutoCAD becoming more or less productive?

Leave a reply

It seems that most of you are convinced that AutoCAD is getting slower, but I’ll leave the poll going for a while longer. But even if AutoCAD is getting slower, does that mean that it’s actually less productive? Do the new features introduced in recent releases allow you to produce more useful work in a given time, despite making you wait from time to time? I’ve added a new poll to see what you think.