Category Archives: Autodesk

Autodesk’s Callan Carpenter responds to Subscription follow-up

5 Replies

You may remember a month ago I raised the question What proportion of Autodesk customers really are on Subscription? Shortly after that, I sent Autodesk Subscription VP Callan Carpenter these questions following up on the interview:

I have a request for follow-up information arising from this interview. I hope you can find the time to provide some answers.

Preamble: Several people have called into doubt your assertion that the simplified upgrade policy affects only a tiny minority of your customers (you seemed to imply a figure of around 3% non-Subscription customers, with 1.5% who upgrade within a year or two). My own calculations based on Autodesk’s latest published financial results indicate that of upgrades represent 21% of the combined income from Subscription and upgrades, which is 7 times greater than the impression you gave in your answer. Please see this post for more discussion.

Questions:

  • Please clarify in as much detail as possible exactly how you arrive at your figures.
  • A percentage is derived by dividing one number by another; what exactly are you dividing by what to come up with 1.5%?
  • Please explain why your statements appear to contradict Autodesk’s own published figures.
  • How large is Autodesk’s total installed base?

Other points of dispute have been raised by various commenters, which I have paraphrased here. I invite your response.

  • Because Autodesk made Subscription cheaper than upgrading, it is no surprise that upgrading became less popular. This doesn’t indicate that customers prefer doing business in that way, merely that Autodesk made it the cheapest alternative.
  • If the idea of Subscription is such an attractive proposition, why do you need to sweeten the deal with tools that you don’t allow upgraders to have?
  • Your assertion that the 12-month cycle is driven by the product teams is incorrect. It was chosen for business reasons and the product cycle was forced to fit the Subscription model.

After a few follow-ups, I received a response yesterday. I reproduce that response here verbatim and without comment:

My sincere apologies for the delay. I have been travelling quite extensively, and this response has been sitting in my drafts email folder, and I just kept getting sidetracked with customer matters.
 
Nonetheless, I appreciate the opportunity to respond to some of the feedback you received after our discussion last month. During that first interview we discussed, among other things, the rationale behind the Simplified Upgrade Pricing program. I argued that SUP impacts only a small subset of our customers, and quoted figures to support the case. It appears those figures have been challenged by a few of your readers who feel their experience is different. Is it possible that both points of view are right? I believe it is.
 
By my prior statements I do not mean to suggest that the vast majority of all customers are on Subscription. Autodesk has a very large base of customers that has grown over the past 28 years. The subscription program as it exists today is only about 8 years old, so we had 20 years to develop a large base of customers, many of whom are not on Subscription. (Yes, there were forerunner programs like VIP, but they were structured quite differently and never generated an appreciable amount of business.) This is important because the SUP program only really impacts those customers upgrading from one and two versions back, which is a very small percentage of the already small upgrade revenue. Subscribers and customers upgrading from four or more versions back see no change to their pricing, and customers upgrading from 3 versions back see either no change or a very nominal one (up or down) depending on their specific product or country.
 
Most of the non-subscribing customer base does not purchase upgrades one or two versions back. In other words, most of these customers either haven’t bought anything from us in a long time, or when they do, they fall into the 98.5% of the revenue that includes upgrades from three or more versions back.
 
History is one thing, but the current trend line is another. For 8 years the Subscription program has coexisted with the Upgrade program. During that time our customers have been free to chose either strategy for keeping their technology current. Based on the results, their choice was clear: the majority of customers buying over the past few years have opted to leverage the Subscription program to stay on the latest technology in the most cost effective way possible. Only a few have elected to stay current through one and two version upgrades. The rest upgraded from older versions – three or more back. Of course Autodesk still offers all those choices going forward, albeit with a slimmed down price sheet.
 
There is one last point that I would like to make: While we believe Subscription is the most cost effective way to stay on the latest design technology, there is much more to the program than cost savings. Direct access to Autodesk product support specialists, Advantage Pack© bonus features, and free software for home use are just some of the value-added aspects of the program. In short, we are committed to an ongoing, continuous reevaluation of both the cost and benefit components of the Subscription value equation in order to make it an attractive option for as many customers as possible.
 
Thanks, again Steve for allowing me the time to speak with your readers.

Missing language pack fixes compared

2 Replies

Having tried out the cleanup fixes from both Autodesk and Owen Wengerd, they both appear to work fine. Here are some points of comparison:

  • Owen’s utility will work with any AutoCAD variant from 2007 on; Autodesk’s fix is currently restricted to Civil 3D 2009, 2010 and 2011. As this problem is definitely not confined to Civil 3D, and may need to be dealt with by non-Civil 3D users, that could be the dealbreaker right there.
  • Owen’s can be installed by anyone by simply copying a file and loading it when needed or in the Startup Suite; Autodesk’s requires admin rights to either run an installer program or manual replacement of a program component, depending on the release.
  • Owen’s loads and runs as the user requires; Autodesk’s runs automatically when opening and saving a drawing.
  • Owen’s provides some information about what is getting cleaned up; Autodesk’s operates in total silence.
  • Owen’s utility can take a while to scan through everything in a complex drawing; Autodesk’s appears to take no longer to open the drawing than normal. To give you some idea of the times involved, in one test in Civil 3D 2011, opening a blank ( but 2.2 MB!) drawing based on the Civil 3D template took 3.6 s with or without the fix; Owen’s cleanup took 0.7 s. In another test on an oldish PC with AutoCAD 2010, cleaning up a drawing with 2.8 MB of real content took Owen’s utility about 15 seconds.

For my purposes, Owen’s utility is what I need, because the users who need to clean up these drawings use AutoCAD, not Civil 3D. I’ve set up a batch process for these users, which opens each selected drawing, runs Owen’s utility and saves the drawing. However, I suggest Civil 3D users install the relevant updates and patches anyway, as they fix more than just this problem. In addition, in Civil 3D 2011 without the Autodesk fix, one of the problems fixed by Owen’s cleanup (a AeccDbNetworkCatalogDef one) is then immediately recreated by Civil 3D.

The upshot is that Civil 3D users should at least apply Autodesk’s fixes; everybody else should use Owen’s.

Using Owen’s fix, it is interesting to see what it reports as being the problem in particular drawings. Here’s what one of my non-Civil 3D problem drawings shows up:

Command: cleanlanguage
Scanning drawing for corrupt objects...
Corrupt object AecDbScheduleDataFormat<2F84> CLEANED
Found 1 corrupt object

Here’s what the Civil 3D 2011 ANZ template shows up when cleaned:

Command: cleanlanguage
Scanning drawing for corrupt objects...
Corrupt object AeccDbNetworkCatalogDef<8B7> ERASED
Corrupt object AeccDbLegendScheduleTableStyle<1619> CLEANED
Corrupt object AeccDbLegendScheduleTableStyle<161B> CLEANED
Corrupt object AeccDbLegendScheduleTableStyle<161A> CLEANED
Corrupt object AeccDbLegendScheduleTableStyle<161F> CLEANED
Found 5 corrupt objects

It looks like every Civil 3D 2011 drawing based on these templates has been going out corrupt in 5 different places. Hopefully, Autodesk will quickly get on to fixing up the Civil 3D template situation, and will incorporate the automated open/save cleanup in future updates to AutoCAD itself and all the other AutoCAD-based verticals.

Another language pack cleanup solution

Leave a reply

My CADLock, Inc. colleague, Owen Wengerd has posted about a fix utility he has written to help clean up drawings infested with the language pack problem discussed here. I have not yet tested Owen’s utility*, but as this should run in any AutoCAD-based product from 2007 on, it could well be a better partial solution than Autodesk’s Civil 3D-only (so far) patches. Autodesk still needs to sort out its dodgy templates, of course, and should probably provide its own non-Civil 3D fixes, if only to maintain a little corporate self-respect.

As Owen has a long and distinguished history of being consistently and demonstrably better at AutoCAD programming than Autodesk’s own programmers, I’d be tempted to try this one first. However, Civil 3D users should probably apply the patches and updates anyway to help resolve other issues.

To find Owen’s utility, go to the ManuSoft ARX freebies page and look for CleanLanguage.zip. While you’re there, use the Software menu to check out some of the other stuff Owen has done.

* Edit: I have now tested it, and it works beautifully in both AutoCAD 2010 and Civil 3D 2011.

Partial fix for language pack problem

1 Reply

The Civil 3D group within Autodesk has moved impressively quickly in providing a partial solution to the language pack problem I described earlier. What has been provided so far is a set of patches for Civil 3D 2009, 2010 and 2011 that allow Civil 3D users to remove the spurious language pack flag by opening and re-saving the affected drawings. I have not yet tested this, but I am informed that it works.

What’s left to do? Obviously, not all recipients of these drawings are going to have Civil 3D. In fact, prior to isolating Civil 3D as one definite source of the problem, I had spent a lot of time helping out AutoCAD users clean up language-pack-infected drawings, using awkward and dangerous copy-and-paste methods. So Autodesk has AutoCAD and all its vertical variants to work through yet as far as a cleanup mechanism goes. Also, the problem needs resolving at the source end. All “infected” templates (in Civil 3D and any other verticals that may have the problem) need fixing and distributing to users as quickly and effectively as possible, in order to reduce the number of drawings being created with the problem. I know individual users can do this for themselves, but large numbers of users won’t do so if left to their own devices, causing problems for everyone else. As the originator of the problem, Autodesk has a duty to do its very best to resolve it.

Thanks, Autodesk, for quickly getting started on fixing the problem and providing a partial solution in a timely manner. I hope you can provide the rest of the solution equally efficiently.

Autodesk discussion group changes – user reaction

Leave a reply

I will be airing my own views on the Autodesk discussion group changes in a future post. In the meantime, I have collected some reactions from other users. For the record, there has been only a little censorship in this area. Here are some of the comments that made it through unhindered:

  • I’ve given it a fair shake and it’s just as bad as I imagined
  • Goodbye, people. It was nice while it lasted
  • it sucks
  • it doesn’t look like you have any intention to meet the expectations of these people
  • not [as] much traffic as there was before the change.  I hope things improve
  • I’m sure you’ve noticed the sourness many folks are having with this interface
  • What a f’in f-up
  • This is so aggravating that I am resorting to posting questions that may have already been answered vs. trying to find them via the search tool
  • Very annoying
  • We use NNTP because it’s easy and fast, and better
  • very slow, compared to “other” html forums
  • This was hyped as a “state-of-art web experience”. It is clearly not
  • Extremely slow compared to the previous web forum
  • we are screwed with this interface
  • This is like having your high performance vehicle (NNTP) stolen and having to take the bus to get to your destination
  • 4 days later, still sucks
  • Still very slow, cumbersome, difficult to track and navigate, unintuitive
  • It took me literally 30 seconds to get that smiley to insert
  • I really was expecting something better
  • I see too many people who may not be around anymore. In most cases their expertise far outweighs any improvements to the forums
  • Welcome to the new and improved Autodesk forum brought to you by high school students near you
  • You keep using that word [“upgrade”]. I do not think it means what you think it means
  • Better? Wanna bet? It’s cumbersome, at best
  • Another annoying thing here is that I can’t seem to find a way to show threading
  • Very, VERRRY slow, compared to forums using PHP and the like
  • Why even have an edit feature at all, when it’s virtually useless?
  • the “experts” are being alienated and having a harder time contributing to aid the beginners
  • I don’t like reading this forum in a browser. NNTP was and is much better
  • This sucks
  • about 40 unanswerable questions that have popped up in the first 30 (wasted) minutes of trying to “give it a chance”
  • If we aren’t going to get our newsgroups back at least attempt to make this forum professional
  • Sorry folks but I just don’t have the time to log in and browse thru all the different pages required now
  • Map 3d is “losing” its best contributor because of a dumb forum update
  • Autodesk prove again if something works  they will find something to make wrong
  • I stopped posting here after the change for the same reasons.  Just logged in for this
  • since the demise of the NNTP feed I rarely visit several of the forums I used to watch
  • It is just too time consuming now. This is really discouraging
  • I cannot be nearly as productive as I could with a newsreader…it takes no less than 4-5x longer
  • getting rid of the NNTP server was one of the worst things Autodesk has done in years
  • Goodbye
  • your update and support policy really force me into alternatives to Autodesk

That’s going down well, then. To be fair, there have been a few people who are relatively supportive of at least some of the changes. As usual with any unpopular change, there are a couple of asinine comments attacking the critics as just a bunch of old whiners who are resistant to all change. But the selection of comments above reflects the overwhelming negative sentiment, and that’s from those people who bothered to stick around long enough to make their views known.

I would have thought Autodesk would have learned its lesson after the well-deserved thrashing it got the last time round, but apparently not.

Censorship on the Autodesk discussion groups

10 Replies

The Autodesk discussion groups have quite a few problems at the moment, which I will discuss at length in future. One unnecessary problem that has been added to the mix is censorship. Having praised Autodesk in the past for allowing discussion to go unhindered, it’s only fair to slam heavy-handed moderation when I see it.

Before I get started, let me just say that Autodesk is entitled to moderate its discussion groups as it sees fit. The forum belongs to Autodesk and it can do what it likes with it. But just because Autodesk can censor its forums, that doesn’t mean it’s always a good idea to do so. Neither does that it mean that Autodesk is immune to public criticism of that censorship. There is no First Amendment obligation on Autodesk, but there are many other places that censored viewpoints can be repeated. Here, for example.

In this particular case, a section was deleted from a reply I made in a thread about the educational plot stamp. In that section, I mentioned that the educational plot stamp is very easy to remove with an everyday AutoCAD command. I didn’t name that command or give any details of how to use it to remove the stamp.

Now I understand that Autodesk gets the twitches when people discuss circumvention of its educational stamp “virus”, but I didn’t mention anything that isn’t already public knowledge. I discussed this issue at length in Cadalyst some five years ago, again without giving away the details. If you really want to know the details, please don’t ask me because I won’t reply. Google it, it’s out there. You probably don’t even need to do that. It’s a pretty obvious thing to attempt. It was, in fact, the very first thing I tried when I first saw an example of an infected file. It worked perfectly.

Back to the censorship. My post was edited, and I wasn’t happy. I wasn’t contacted about it, so it was not possible to have a reasoned discussion about it with the moderator (as I have done in the past on the AUGI forums and elsewhere). Annoyed, I made a further post, this one objecting to the censorship. In that post, among other things, I pointed out that the Autodesk position on the plot stamp was fictional. Here is what the Autodesk knowledge base item TS63668 (which I can no longer find) had to say on the subject:

Issue
When you plot a drawing that was created in or that contains drawing data that was created in the Educational (Student and Faculty) version of AutoCAD® or AutoCAD-based software, the following plot stamp or watermark appears in the plot:

For Educational Use Only

Solution

There is no way to circumvent the plot stamp. This is as designed to discourage the commercial use of an educational version of an AutoCAD product. Autodesk sells educational versions of software on the premise that the software will be used for educational purposes only.

The statement above in italics is a blatant lie. Hopefully, the knowledge base item is now missing because somebody sensible at Autodesk decided that it’s not a good look to have such fraudulent nonsense on its site, dishonestly masquerading as technical support. Or maybe it’s not missing but I can’t find it because the search engine is bad. After all, Autodesk really, really sucks at search. Perhaps it should buy a search engine company?

I digress; back to the censorship issue again. My post objecting to the first censorship was deleted. I was not contacted to discuss this deletion. I made another post objecting to the second censorship of my objection to the first censorship. This post made no reference whatsoever to the plot stamp issue itself. This post was deleted, too. In a surprise development, I was not contacted to discuss this deletion. Three levels of censorship to cover up an Autodesk lie. I can’t see a problem with that, can you? Except for this:

The Internet interprets censorship as damage and routes around it. — John Gilmore

Discussion_Admin, you were entirely within your rights to perform this censorship. Your moderation guidelines may even require it. But as a result, my statement about the plot stamp being easily removed has been read by a much larger number of people. So it really wasn’t such a good idea to censor it, was it?

Readers, if you have your own Autodesk censorship tales to tell, feel free to tell them here. It should be a fun read.

More Autodesk research – Groups

2 Replies

Here’s an announcement from the AutoCAD Product Design & User Experience Team:

AutoCAD User Research Study: “AutoCAD Groups”(AutoCAD Group command)

AutoCAD Product Design & User Experience Team is looking for your input regarding the AutoCAD GROUP command usage.

The GROUP command (Object Grouping Dialog) in AutoCAD allows creating a selection set of objects called a group.
When an object belongs to a group, if any object in the group is selected, all the objects in the group are selected.
Groups can be named or unnamed. Groups can be ungrouped/(exploded), which removes the relationship between the objects in the group.

Autodesk wants to better understand how you use Groups so we can improve the feature.

http://www.surveymonkey.com/s/AutoCAD_GroupCommand

Go to it, people. It’s good to see some attention being given to some long-neglected parts of AutoCAD. What next, LISP?

Autodesk discussion group update – what do you think?

11 Replies

On 4 June 2010, Autodesk turned off NNTP access to its discussion groups as part of the process of updating its software to use a different engine (the new one is from Lithium – here are its own forums). I am preparing a large post about what I think of the new web interface, but for now let’s hear from you on that subject. Please vote in the poll on the right, and add your comments once you’ve had a chance to put the “state-of-the-art web experience Autodesk customers have come to expect” through its paces.

In related news, I have now closed the short-lived poll about the end of NNTP access to these groups. The results were:

Should Autodesk shut down NNTP access to its discussion groups?
Yes (8.8%, 5 Votes)
No (59.6%, 34 Votes)
Don’t care (31.6%, 18 Votes)
Total Voters: 57

This is a small sample and must have some self-selection bias, in that those who cared about this move were more likely to read my post on the subject and vote about it. I attempted to temper this by including a “Don’t care” option, but some bias is still bound to be there. There is also likely to be some bias in the opposite direction, because people are less inclined to bother voting to try to fight a decision that had clearly already been set in concrete and which was never going to change.

That said, it does seem remarkable that only 5 people could be found who supported Autodesk’s decision to drop NNTP access. According to my long-running What is your relationship to Autodesk? poll, There are at least 25 (claimed) Autodesk employees who are active enough on this blog to respond to its polls! Without wishing to compromise the private nature of my polls, I can reveal that the 5 Yes votes included Autodesk employees and at least one non-Autodesk person (that’s as specific as I will ever get). There did not appear to be any attempt to distort the voting from either camp. I mention this because the survey mentioned in my previous post was disrupted in this way.

What proportion of Autodesk customers really are on Subscription?

7 Replies

In my recent interview of Autodesk Subscription VP Callan Carpenter, he made these statements:

…there is a very small fraction of our revenue that comes from upgrades at this point in time.

We’re down to very low single digits of customers who upgrade, and of those only half of those upgrade 1 or 2 years back. So we’re talking about approximately 1.5% of our revenue that comes from customers upgrading 1 and 2 versions back.

…[customers who upgrade] 1 or 2 [releases] back, a very small percentage of our customer base, less than 2% of our customer base that was buying those upgrades.

Others are calling those numbers into doubt. Deelip Menezes (SYCODE, Print 3D) estimated the numbers of AutoCAD users not on Subscription at 66% (or 43%, depending on which bit of the post you read), by counting the AutoCAD releases used by his customers and making assumptions about their Subscription status from that. That’s an extremely suspect methodology, as I pointed out:

Your numbers don’t really tell us anything about Subscription v. upgrade proportions. All they tell us is that large numbers of people wait a while before installing a new release. We all knew that, surely.

However, Deelip’s post did prompt me to point out this:

…there is a fair point to be made about people on earlier releases who have hopped off the upgrade train altogether, or at least for a significant number of years. How would they be counted in Callan’s figures? They wouldn’t exist at all, as far as his income percentages are concerned.

Owen Wengerd (ManuSoft, CADLock) asked a random sample of his customers and came up with 82% of them as non-Subscription customers. He also noted that he could come up with a 3% non-Subscription figure if he cooked the books by selectively choosing a convenient time slice. Owen doesn’t state the numbers in his sample, or indicate (or know) how many of the non-Subscribers are also non-upgraders.

I’ve added my own poll (see right) just to add to the mix.

Nothing we can hang a conclusion on yet, then. But Ralph Grabowski (WorldCAD Access, upFront.eZine) uses Autodesk’s own figures to point out that upgrade revenue has increased 18% and Subscription revenue only 7% in the last year. I’m not qualified to perform an analysis of the 2011 Q1 fiscal results, but I can find the figures listed as Maintenance revenue ($195 M) and Upgrade revenue ($51 M). That looks to me like about 21% of the Subscription/upgrade income is coming from upgrades.

Also, according to the published figures, Autodesk has 2,383,000 customers on Subscription. If that represents about 97% of customers, does that really mean Autodesk has only about 2.5 M customers? If I’m looking at these figures in the wrong way, feel free to put me right.

So, what’s the truth? What proportion of Autodesk customers really are on Subscription? 3%? 21%? 43%? 66%? 82%? I’m going to ask Callan a follow-up question about this and will report back on what he has to say. In the spirit of this post, I’ll be asking him for a lot more detail. Watch this space.

Autodesk to kill NNTP discussion groups

10 Replies

As of 4 June, Autodesk intends to update its discussion group software to something that does not support newsgroup (NNTP) access. From an email by Autodesk’s Eric Wright to NNTP users:

“As an active NNTP user, we wanted to reach out to you directly. We recognize this will change your experience participating in the forums and want to help you transition to the new web interface. Improvements include a simpler, more intuitive interface to post & reply, bookmarking and e-mail notification features to track favorite posts, and more powerful search tools and filtering. While not a substitute for the NNTP experience, the streamlined capabilities of our enhanced RSS feeds can also provide an alternative offline forum reading experience.”

As you can see, we are significantly investing to improve the platform behind the web-based experince to address many of the shortfalls reported by users over the last few years. Rich text vs Plain text confusion, formatting issues (like I just experienced cutting and pasting this message), logout issues, search, in-line image support, and robust RSS capabilities are just a few areas of improvements in an update planned for June 4.

A public announcement will be posted in the forums in a few days. I hope you will give it a try after launch, and provide any feedback or best practices to help in the transition.

Eric Wright

Product Manager – Support & Learning
Web & eBusiness
Autodesk, Inc.

The public announcement mentioned above can be found here.

As you might expect, this decision has been a hot topic of conversation. A survey has been set up (by Tony Tanzillo, not by Autodesk), and the running results are here. I’ve added a poll of my own (on the right). Feel free to express your views here, too.

I have some sympathy for Autodesk in this situation. One of the reasons the disastrous discussion group update of 2008 bombed so badly is that Autodesk was restricted in what software was available that supported both NNTP and web access. By taking the decision to dump NNTP, there is a much better chance of providing a system that works adequately (although Microsoft appears to be able to manage both). Whether an adequate web forum system actually happens or not remains to be seen, but I can understand the thought process that would lead to the decision, which Eric admits was “difficult and bittersweet”.

On the other hand, I am in no doubt that this is going to hurt the discussion groups. I don’t have any figures on the proportion of users that use NNTP, but I do know that a very significant number of the most active and expert users use NNTP. They do this because it’s vastly more efficient to work that way when dealing with large numbers of messages. Occasional users like myself are content enough to hop in from time to time and browse around using the web interface, maybe answering a question or two. The people who live on there, the people who are the groups’ primary resource as a free-to-Autodesk support mechanism? NNTP users, mostly. And what’s the point of a self-help group without a knowledgeable community of people to do the helping?

Adobe went through something similar a while back (links courtesy of CAD Panacea). I don’t know how many good people Adobe lost or how many Autodesk is going to lose now, but I know it’s going to be greater than zero. It will be interesting to see how useful the Autodesk discussion groups are after this change, and not just in terms of the interface and access to existing content. How useful are they going to be as a place to ask questions and stand a chance of getting a knowledgeable answer? I know Autodesk has been experimenting in having some support people respond in the new Installation & Licensing group. Maybe that’s the plan for the future? Time to start hiring back some of the 10% of people Autodesk lost early last year?

Does Autodesk discuss future plans?

6 Replies

According to Shaan, Autodesk does not discuss its future plans. Or does it? In a comment, Ralph reckoned it does. Putting aside technology previews and various NDA-bound circumstances (e.g. Beta testing), can you think of cases where Autodesk has revealed what it intends to do in future? Here are a few off the top of my head:

  • I’ve been to AU sessions dating back to 1995 that pretty much give away the contents of the next release of AutoCAD, using a vague cover-my-butt session title and a disclaimer at the start of the session. I understand that these days, attendees need to sign an NDA before entering such sessions.
  • Last year in San Francisco, an international blogger audience was given all sorts of information about Autodesk’s future directions (preceded by a similar disclaimer), with no NDA and nothing off the record. I assume something similar happened at this year’s North American bloggers’ event.
  • The Subscription (Advantage) Packs of the last couple of years have been a dead giveaway about some of the features that are going to make their way into the next release.
  • The new 50%-cost upgrade policy was announced a year in advance.
  • Surveys and other customer feedback mechanisms provide a very big clue about what Autodesk is looking at changing next. Some of these are covered by NDA, others are not.
  • In the specific case that triggered this discussion, Autodesk has been gradually building up expectation of a Mac AutoCAD for quite a while. Yes, it required a little reading Between the Lines, but for some time it has been pretty obvious where all the Mac love was leading to.

Feel free to add your own examples, but it seems to me that Autodesk is perfectly happy to reveal future plans as and when it sees fit. And that’s fine. In each of the above cases, the revelations have been A Good Thing. Good for Autodesk, good for customers.

Maybe the question should be, “Why doesn’t Autodesk discuss future plans much more often?” Stock market regulations, perhaps? But hang on, there are some very major publicly traded corporations that seem to get away with revealing all sorts of things about their future products. For example, Microsoft regularly conducts very widespread public Beta programs that let people know in great detail what’s very likely to appear in the next release, and seems to have survived the experience so far. There’s surely no reason why Autodesk couldn’t do the same if it wanted to.

Ultimately, it comes down to a desire for secrecy; a culture of concealment and control. Of course Autodesk may have legitimate reasons for keeping some of its plans from its competitors, but the culture can be so pervasive as to cause some bizarre side effects. You may find this difficult to comprehend, but there are those in Autodesk who got into a tizzy about me speculating in my launch announcement that Autodesk’s general design product (AutoCAD 2010) was going to be followed by something called AutoCAD 2011. There was something of a surreal drama behind the scenes. There were apparently people within Autodesk who genuinely thought I needed privileged information to work out that 10+1=11. No, I’m not making this up.

I’m not sure Autodesk’s secrecy is doing any good for anyone. It’s surely harder to maintain these days and it’s only going to get harder. I suspect several Autodesk blood vessels were burst when AutoCAD Mac Beta 1 was leaked. On the one hand, I can understand that; somebody broke an NDA, a legitimate agreement was freely entered into and then broken. Some people at Autodesk probably had their carefully planned marketing timelines disrupted.

On the other hand, this provided a whole heap of free and largely positive publicity. Potential AutoCAD for Mac users are now hovering in anticipation, filling the Mac forums, spreading the good news among themselves, putting off the purchase of competitive products, considering entering the official Beta program, and so on. At the same time, the news of performance issues in the early Beta is helping to put a dampener on expectations in that area. Lowered customer expectations may turn out to be very useful when the product is actually released. All considered, a good thing for Autodesk, then.

I’m convinced that Autodesk is opening up. That’s great, but there’s a way to go yet.

Do you think Migration sucks?

2 Replies

I do. If you’ve added a couple of toolbars and changed a few settings, it’s probably fine for you. But I think it’s been effectively broken for significantly customised setups ever since Autodesk “improved” it by introducing the CUI mechanism in AutoCAD 2006. It’s undocumented and whenever I’ve tried it, unreliable. I ran some polls on it a couple of years ago which had few responses. What do you think now?

If you’re unhappy with migration, don’t just vent here. Autodesk now wants to hear from you. Here’s the announcement:

Dear AutoCAD User!

AutoCAD Product Design & Usability Team is looking for participants for the study.

Topic: focus on Migration process, Migration tool and results of migration.

Our Goal

To gain the most complete understanding about problems and requests AutoCAD users may have while migrating their settings and customization from a previous release of AutoCAD.

Who Should Participate?

We are looking for individual contributors or CAD managers with small number of seats (less than 5- either standalone or multi-seat standalone) with unsatisfying experience using Migration tool to migrate settings from a previous version of AutoCAD.

How the Study will be Conducted?

We will schedule ~1 h interview session with you (remotely) and discuss your experience with migration, results you expected, outcome you’ve got.

When?

We are planning research between May 27 and June 2, 2010.

How To Sign Up?

Please submit qualification data and indicate your availability here:

http://www.surveymonkey.com/s/MigrationSignUp

I’m a bit concerned about the restriction of this study to individual users and CAD managers with a handful of users, as I would have thought CAD managers with significant numbers of users would be the least satisfied group, and the group with the greatest need for a working Migration system. However, as with other such Autodesk research, I encourage your participation.

AutoCAD for Mac in Beta

4 Replies

Disclaimer: I have absolutely no access to inside information about this Beta. Even if I had, I would not reveal anything that I had learned as a result of such access. This post discusses only information that is already public knowledge.

The native Mac OS X AutoCAD port that Autodesk has been foreshadowing for some time is now in Beta, it seems. The Italian Mac community is getting particularly excited about the leak, but it’s also a popular subject of discussion on at least one English-speaking forum. The Autodesk codename is Sledgehammer, and it’s currently 64-bit only. If this is a subject that interests you, with a bit of sniffing around you can easily find screenshots, a video and you can apparently even download it via torrent if you’re feeling particularly brave/stupid.

If you’re interested in trying it out, it would be much better to apply to join the Beta program. That way, you will stay legal, you won’t download a trojan and you will contribute towards improving the product. Autodesk will probably need such contributions, because the early Beta allegedly runs like “a sewer” with huge performance issues. That should not be a surprise at this stage, but it should give you some idea of how much work Autodesk has ahead of it before it has a product that is fit for human consumption.

Oh, if you do join Autodesk’s Beta program, please be a bit more careful with the software than the guy who thought it would be a cool thing to hand out to his friends.

Edit: Ralph thinks it’s fake. I really don’t think it is, but must acknowledge the possibility that I’m wrong.

Edit 2: More discussion and screenshots at SolidSmack.

Callan Carpenter interview 5 – the 12 month cycle

2 Replies

This 5th post concludes the Callan Carpenter interview series. For the record, this interview was done in real time over the phone, with no prior notice of the questions.

SJ: The 12-month cycle that you have for most of your software has come under some criticism from all sorts of people, especially me. Once you have your customer base practically all on Subscription, what’s the incentive for the 12-month cycle to persist?

CC: In what way have you criticised the 12 month cycle?

SJ: In that it damages the product. In that there’s not enough time to release a properly developed product within that 12-month cycle. This is an observation that many people have made going back many years. That’s the basis of the criticism; not that, “Oh no, you’re giving me more software”. Well, there are people who complain about that but I don’t think that’s a valid criticism. I think the valid criticism is that it damages the product. A poll that I ran on my blog asked that question: is the 12-month cycle damaging the product? The answer was a very emphatic yes from the readers of my blog. I know that’s not a scientific survey but it fits in with other viewpoints I’ve seen expressed in various places.

CC: The question was, do we intend to continue to do that?

SJ: Yes. Once you have effectively have your customers on the Subscription model, so that you’re no longer internally competing with the upgrade model, do you really have to have a 12-month release cycle?

CC: Well, I think it’s a very interesting and valid question, do we need to have a 12-month upgrade cycle? I know there are customers who simply cannot absorb technology at that rate. But it’s a bit of a two-edged sword, in that if we go to a 24-month cycle, for example, do we get criticism for not providing enough value for the Subscription dollar or is it going to be viewed as a positive because it’s improved overall software quality? If we stay at the 12 months, we get the reverse argument. Maybe we’re providing the value that customers are paying for with Subscription, but what are we doing to software quality? I think that one of the things we have to look at over time is alternative delivery mechanisms. You’re going to start to see, for example, software delivered (as we have started to) with things available as Software as a Service. That obviates a lot of the issues associated with those release cycles you’re talking about. Your quality can go up, it’s a lot more controlled environment, and the customer doesn’t have to deal with an install, then another install and another install. So I would imagine you would see augmentation of our desktop products with products like that, that sort of move away from the complexities of the constant need to try and absorb new technology.

I think that it would be a very interesting thing to do on a scientific basis to understand whether customers prefer us to go a 24-month or an 18-month, or you-pick-the cycle. I think internally, your question about is it motivated by some kind of internal competition with upgrades, absolutely not. Upgrades, just look at the numbers, that battle’s over, so there’s no internal competition in that regard. The thing that we do have to deal with, which I think is endemic to any engineering creative group, is software engineers like to write software. They’re not motivated by issues of Subscription, or upgrade, or anything else. What they do is create product. We would literally have to rein those guys back if we wanted to go to a longer cycle. They’re the ones leading the charge on that, not the Subscription program.

SJ: So you’re saying that the development teams like the 12-month cycle?

CC: They do. It brings a certain discipline to them on the one hand; on the other hand, it’s kind of what software writers do, they write software.

SJ: Right, but they can write software that takes 12 months and isn’t finished or they can write software that takes 18 months and is finished. If I were a developer I know which I’d prefer.

CC: I hear your point. I think something we have to always look at is what’s the right balance between functionality and trying to build a bridge too far and to get it released. That’s something I know the product division managers are looking at constantly. Again, it’s absolutely not motivated by Subscription. Like you, I’ve heard customers say, “Would you go to 24 months?”, so I’d be happy to deliver that for them in some cases. But it’s really up to the product divisions.

See also
Callan Carpenter interview 1 – Autodesk and social media
Callan Carpenter interview 2 – upgrades a tiny minority
Callan Carpenter interview 3 – the cost of complexity
Callan Carpenter interview 4 – enhancing the program

Callan Carpenter interview 4 – enhancing the program

1 Reply

Part 4 of 5 in this series.

SJ: There is always the fear that once you have all of your customer base on Subscription, you’re not going to need to offer those benefits any more. Can you assure people that that’s not going to be the case, that you are going to keep being “nice” to your customers?

CC: Absolutely. I think my team and I spend as much time and brain energy trying to figure out how to enhance the program as anything else. Our goal is to make Subscription a compelling value proposition; to make it not only cost-effective but valuable in other ways. An example would be the Advantage Pack program. We had a history of Subscription including extensions and other little technology bonuses for subscribers. But last year, we said we’re going to do something different with that. One of the problems with our historical technical Extensions program is that it was optional for product line managers to either participate or not. It was optional for product line managers to localise those Extensions in languages other than English. It was optional to make those Extensions incremental install as opposed to requiring a full reinstallation of a product.

So last year, we turned a lot of our product development upside down and produced the Advantage Pack with a whole new set of requirements. A product had to be localised, it had to be incremental install, and the top 26 or 27 product lines all had to participate in delivering that value. We saw the impact in the form of a 150% increase in the downloads of that Advantage Pack. That’s an example of trying to improve the value, and you’re going to see some additional fairly significant moves on our part on the Advantage Pack this year that are going to have a lasting impact on Subscription and how people look at it. Next year, we plan to improve it yet more, and so on. I don’t see any end in sight. It’s a competitive world out there and the only way you survive is by continuing to improve and grow and add value, otherwise you get replaced, and nobody’s immune to that. No company, no market.

SJ: You said that there was a 150% increase in Advantage Pack downloads. What was increasing over what?

CC: Prior to Advantage Pack, we had the Extensions program. If you took all the Extensions for the various products for the prior year, the last year we had Extensions (2008), and you compare that to the number of downloads of the Advantage Pack, it’s a 150% increase of Advantage Pack downloads over Extensions.

SJ: But there had been no Extensions for AutoCAD since the very early years, right?

CC: No, there were Extensions for AutoCAD. For example, there was an AutoCAD Extension 2 years ago for PDF writing. This year, I don’t want to give the cat away, but you’re going to see some very interesting technology that is being made exclusively available to subscribers for no additional charge, that I think they will find quite interesting.

SJ: I was scratching my head to think of Extensions; after the initial burst when they first came out there was practically nothing. There was a trickle of them that came through for the various verticals, but I’m struggling to think of AutoCAD ones between, say, 2002 and 2007.

CC: I think you’re highlighting a potential example of a challenge that we had with the old Extensions program in that it was optional for product line managers to participate or not. Today, that’s not the case and that includes AutoCAD. They do participate in the Advantage Pack program and will continue to do so, along with Revit, Inventor and 20-odd other products that are our biggest sellers.

SJ: So that’s a permanent fixture as far as you’re concerned? The Advantage Packs aren’t going to disappear?

CC: Not unless we can come up with something better.

SJ: So there are no other nasties you have planned for customers? You’re not going to, for example, change the EULA so that Subscription is going to have to be paid otherwise your licenses don’t work any more?

CC: No, at this stage we don’t see any change to the perpetual license model if that’s what you’re referring to. We have a hybrid model, which is different from some industries. Some industries are all perpetual, some industries are all term-based licensing, we are still perpetual, plus Subscription or maintenance. I don’t see that changing. It’s hard to predict 50 years into the future, but we have no plans for that.

See also
Callan Carpenter interview 1 – Autodesk and social media
Callan Carpenter interview 2 – upgrades a tiny minority
Callan Carpenter interview 3 – the cost of complexity

Callan Carpenter interview 3 – the cost of complexity

2 Replies

Part 3 of 5 in this series.

SJ: In one of my blog posts, I was pretty cynical about one of the phrases used in the press release: “the streamlining of upgrade pricing based on feedback from customers and resellers”. Was I wrong to be cynical about that? Did your customers really ask for upgrade prices to be increased to some nice round number?

CC: What our customers have asked for is simplified purchasing. We have a very complex price book and it leads to thousands of prices items, maybe tens of thousands when you have all the permutations across all the different geographies in which we sell software. A lot of that complexity came from having multiple-step upgrades, multiple-step crossgrades. There is a cost to maintaining that kind of a system. So our resellers certainly were asking for simplification and streamlining explicitly. Our customers were asking to find ways to make it easier to do business with Autodesk; can it be less expensive? One of the costs of doing business is maintaining a very complex pricing scheme as we have in the past. So while we may not have a customer say, “Gosh, I wish you would simplify your upgrade pricing” explicitly, it is implicit in trying to offer an easier path to buying and less cost in the long run because we’re not maintaining a very complex system that only serves a very small percentage of our customer base.

SJ: So there’s a real cost associated with this. Can you put a number on that as a percentage of the cost of the upgrade? Is it 1%? 10%? Is a big amount that customers need to be worried about?

CC: You know, I’ve never tried to put it as a percentage of the cost of an upgrade and tried to figure it out. Some of these things are a little difficult to untangle, but you can look at the complexity of your back office software, the staff that it takes to maintain it, the cost of the releases; our customers are simply aware of our releases of our software products, but they’re not aware of the fact that of course we have numerous releases of our internal systems for tracking and matching assets, price books, things like this. All of those have a cost associated with them. People, software, systems and so on. I haven’t ever tried to calculate that as a percentage of the cost of an upgrade, but it’s certainly a real cost nonetheless.

SJ: So let’s say it was costing people 5%. Why didn’t you reduce the prices by 5% instead of trebling them?

CC: Well, we didn’t really treble the prices. What we did was we said, remember for 3 or more versions back, the price is essentially unchanged. It may be a couple of hundred dollars more expensive or less expensive depending on the product and the market for the third version back. The big change was really in 1 or 2 back, a very small percentage of our customer base, less than 2% of our customer base that was buying those upgrades. It didn’t really make sense to us to maintain the complexity for that small percentage of our customer base.

I think that it’s an interesting point that we’re in because if you go back far enough in time, and you don’t have to go back that far, about 8 or 9 years I guess, with Subscription we could have been arguing the other extreme. We could have been arguing that, “My goodness, why are you making me pay for upgrades?”, and this Subscription thing either didn’t exist or it was very, very expensive. And then we designed Subscription to actually be very cost-effective, to be the most cost-effective to get access to this technology. So it’s an interesting inversion. I think it would be an interesting mind-experiment to wonder what would happen if we took away our Subscription pricing tomorrow, which is typically somewhere between 10% and 18% of list price of the product, depending on the product and the market. If we took that option away tomorrow, I think we actually would create tremendous havoc in the marketplace, because that’s really where the majority of our customers are today in terms of buying our software.

SJ: There are people who do still want to buy upgrades, those who want to have that choice. Do you understand the mindset of people who say, “I want to see what the product is before I pay for it”?

CC: I can appreciate that sentiment. I’d like to believe that our 25+ years of history has generally shown that our pace of advancement is generally up and to the right. Certainly there have been hiccups along the way; some releases have more functionality than others, but generally it’s up and to the right. But the customers who wish to do that, I certainly can appreciate that and that’s as good a reason as any for why we’ve kept upgrade and Subscription pricing as opposed to one or the other, because it gives customers a choice. For those customers who tend to want to wait and see, again the vast majority of them are doing it 3 or more versions back. If they’re doing it less than that, they’re on Subscription, by and large. So they still have that option. Even with the simplified upgrade pricing, I think it’s important to point out that we announced it over a year ago, and even today, if a customer goes off Subscription, they have up to a year to retroactively attach it. So the hope is with that timeframe, questions of, “Is the economy going to turn back up?”, those sort of things will be answered. If it takes 2 years for those questions to be answered, well then you’re back to 3 versions back pricing or more, and that has hardly changed, if at all. So I think that those kind of customers that want to wait and see what the product is going to be before they buy it, they have that option.

You have to also realise that there are also other benefits that come from Subscription in addition to the upgrade. Access to our product support teams, access to prior version usage, home use licenses, the prerequisite to global floating network licenses and other types of benefits, those are a very significant proportion of the value.

See also
Callan Carpenter interview 1 – Autodesk and social media
Callan Carpenter interview 2 – upgrades a tiny minority

Callan Carpenter interview 2 – upgrades a tiny minority

5 Replies

Part 2 of 5 in this series.

SJ: Is there anything specific you want to say about what I have written in my blog?

CC: There are a number of things we can do to put Subscription questions and Simplified Upgrade Pricing into context. I think the first thing we need to recognise is that there is a very small fraction of our revenue that comes from upgrades at this point in time. For the last 8 years or so, our customers have fairly well self-selected to either prefer to be on Subscription and have the latest version and technology available to them, or to not do that, in which case they tend to upgrade 3 years or more after the current release. We’re down to very low single digits of customers who upgrade, and of those only half of those upgrade 1 or 2 years back. So we’re talking about approximately 1.5% of our revenue that comes from customers upgrading 1 and 2 versions back. And so I think there’s clearly been a natural selection, a natural fallout over time of customers choosing; do I prefer to be on Subscription or do I prefer to pay for an upgrade?

If you look at the real impact of upgrade pricing, the real impact is the customers who prefer to upgrade from 1 or 2 versions back, that’s a very very small percentage of our business. For those who are 3 versions back or more, there’s really no change at all. For subscribers, which is the majority of the customer base, there is no change at all either. I just wanted to start by kind of putting that in perspective.

I think the other thing we should look at is that the history of the Subscription program is one of actually creating more value over time. It started out as simply an upgrade path, a cheaper path to upgrade than buying upgrades. Over time we’ve added more value in terms of additional support options, additional licensing benefits that come with Subscription and later on this year you are going to see things like a very enhanced Advantage Pack program, which started last year.

So as I read through a lot of the blogs, I was struck by a kind of lack of perspective on how the program has grown over time and how very few of our customers were actually buying upgrades.

SJ: There are some of your customers that don’t have any option but to be on Subscription, aren’t there? There are some markets and some products where Subscription is compulsory, right?

CC: No, with a few exceptions, I don’t believe we have any compulsory Subscription left. There may be a few in some emerging countries where software piracy is a particular issue, but generally speaking, the vast majority of our customers have the option to either be on Subscription or not.

SJ: For some years here in Australia, if you wanted to upgrade to the latest release, Subscription has been compulsory. Is this unique to Australia or does this happen elsewhere?

In Australia we do have a unique experiment, but that is fairly unusual. No other country comes to the top of my mind.

SJ: Is this experiment going to continue or does the point become moot now that the price of upgrading has been increased?

CC: I don’t think we’re going to be changing the way we do business in Australia.

See also
Callan Carpenter interview 1 – Autodesk and social media

Callan Carpenter interview 1 – Autodesk and social media

Leave a reply

A couple of weeks ago, Angela Simoes from the Autodesk Corporate PR team invited me to interview Callan Carpenter, Autodesk’s Vice President of Global Subscription and Support. Callan is responsible for the sales, marketing operations and product support associated with Subscription. He is also Vice President in charge of Jim Quanci’s Autodesk Developer Network. This morning, we had a very extensive discussion about Subscription and other topics that I intend to publish in several parts over the next few days. Deelip has already published a Callan interview, but mine is quite different.

In this post, I will let Callan introduce himself and then move into some questions about social media that I asked at the end of the interview. In this post, both Callan Carpenter (CC) and Angela Simoes (AS) responded to my questions.

SJ: Callan, can you give me some background on yourself?

CC: I’ve been at Autodesk since November 2008. Prior to that, I spent 20-odd years in the semiconductor and semiconductor-CAD software business: technologies in many ways analogous to what we have for our manufacturing, civil and media/entertainment markets here. I was focused on semiconductor design, manufacturing, electrical properties and so forth. I’m an electrical engineer by training. I’ve spent about half of my time in startups and about half in big companies. Everything from designing silicon to sales and marketing to engineering to you-name-it.

SJ: It’s kind of unusual for me as a mere blogger to be approached by a Vice President, but I’ve had this happen twice in the past couple of weeks. Is there a move within Autodesk to engage more with bloggers and social media?

CC: We’re definitely more conscious of social media than we have been historically. We are becoming more cognisant of the power of social media, whether it’s tweets or blogs or other forms. Like any company we have to adapt to that, respond to that and participate in the conversation.

AS: There’s no doubt that the line between what you would call traditional media and social media or bloggers is really blurring, and has been blurring over the last 5 to 10 years. You can’t deny that there are some bloggers, like yourself, Steve, who are quite influential in their industries. So it’s a natural move for us to start engaging more closely with bloggers, especially the ones that are clearly using our product every day, have a very engaged audience, who are really discussing some meaty issues on their blogs. Because we want to ensure that you have just as much information and access to our executives as someone in the traditional media would. Yes, we’re absolutely engaging more closely with bloggers.

SJ: Is this a policy decision or has it just naturally happened?

AS: It just sort of naturally happened. I’ve been here for 4 years now and ever since I’ve been here we’ve always engaged with bloggers in some capacity. This has increased over the past 2 or 3 years, significantly.

CC: I think it’s fair to say that Carl, our CEO and Chris Bradshaw, our Chief Marketing Officer, are very cognisant, very sensitive to… we have to adapt to the way our customers, and our next generation and next generation of customers are using technology, using social networks. They challenge us to not stay stuck in the old paradigms. So there’s a lot of support from the top for engaging in social media.

AS: I think you’d be surprised how many people at the top actually read various blogs and follow what people are saying. They definitely pay attention.

SJ: I read an interesting article the other day about who should be running social media for a corporation; should they have a specific department for it or whatever? How is that happening in Autodesk? Is that a PR function or does everybody do it?

AS: We as a company across all departments have put a lot of effort into looking at what other companies are doing, what works for large companies, trying to find the model that fits best for us. We found that there were already a lot of people across departments participating in Twitter and Facebook, posting videos on YouTube, and so the coordinating functions will sit within marketing, but each department or industry division will have a representative on a social web council, where we are collectively making decisions together. But there isn’t one person doing all this.

CC: So for example, in my capacity with product support, there is an element of social media to our strategy there as well, starting with an improved set of forums and new methodology there that we will be unrolling there later this year as part of our new comprehensive remake of our self-help infrastructure. Then I don’t know where it’s going to evolve to, I think that’s one of the interesting things about social media is, who knows where this is going to end? But we’re definitely looking at it from a support perspective, a marketing perspective, a PR perspective, even from a sales perspective, so there’s so many different dimensions to it. It’s coordinated by marketing, but it’s starting to enthuse almost all aspects of the business and that’s a very interesting thing. Who could have predicted that 10 years ago?

AS: We want to make sure if someone is tweeting about a problem with a download, or somebody isn’t able to log in to the Subscription Center for some reason, or even can’t find information on a product they’re looking for, all of those are different problems to be solved by different groups, and so we are putting into place listening mechanisms so that we are listening to our customers and addressing their concerns and questions in a timely manner. And that does take a lot of coordination internally but we’re working hard at it.

CC: A good example was over a year ago, we did our first major software download for the upgrades in 3 trial countries. One of the feedback mechanisms we used to improve the design of the download experience was listening to tweets.

Autodesk’s CIP data – massively biased?

10 Replies

You may have seen Shaan Hurley and I having a discussion (ahem) over the validity of his statement:

I really do use the ribbon now with AutoCAD 2010 along with most users as evidenced by the CIP data we receive daily from thousands of AutoCAD users who choose to send the great data.

So, now you know. Most of you use the Ribbon now,  Shaan said so. Shaan, as he always has done in the past, declined my invitation to back up this assertion with more details. He has vast amounts of data collected from huge numbers of users. How could that possibly be wrong?

Here’s how. CIP data is biased.

How can millions of data points be biased? Actually, all samples are biased. Only the degree of bias varies. The polls on this blog are no exception. I do my best to keep the questions and options neutral; the only leading questions you’ll see here in serious polls are the ones I copy and paste from Autodesk blogs. But readers of this blog are one self-selecting small portion of Autodesk customers, and people who vote in my polls represent another self-selecting portion of that portion.

The question is, how biased is Autodesk’s CIP data? Without access to Autodesk’s data (which it won’t provide) and resources for alternative data collection from its customers (ditto), the best I can do is use my own biased sample (that’s you lot out there) as a cross-check.

Let’s examine it in light of Ribbon use among AutoCAD 2010 users. In an earlier comparison of my 2009 poll figures and Shaan’s CIP data, I wrote this:

But Shaan’s CIP users are also a biased sample, comprising those AutoCAD users who have CIP turned on. Are users who go with the flow and have CIP on also more likely to go with the flow and leave the Ribbon on? Possibly, but I would have thought the CIP-on bias would be less significant than the blog-reader bias.

I have recently run a poll to try to determine if that “possibly”, that hunch, has any basis. Let’s examine the results I got.

AutoCAD 2010 users, what are your Ribbon and CIP settings?

Ribbon on, CIP on (24.7%, 65 Votes)
Ribbon on, CIP off (19.4%, 51 Votes)
Ribbon off, CIP on (11%, 29 Votes)
Ribbon off, CIP off (44.9%, 118 Votes)

Total Voters: 263

For the sake of argument, let’s make the assumption that my poll sample is unbiased. It’s not, and the degree of bias is unknown, but let’s see what it would mean if it was. Let’s see what kind of results Autodesk would see from its CIP sample:

CIP-on voters (94):
Ribbon on 69% (65)
Ribbon off 31% (29)

Shaan would see from this result that 69% of AutoCAD 2010 users have the Ribbon on, and would be tempted to say stuff like “use the ribbon now with AutoCAD 2010 along with most users”. Understandable. That’s just CIP users, but non-CIP users can’t be that different, surely? Or can they?

CIP-off voters (169):
Ribbon on 30% (51)
Ribbon off 70% (118)

Wow. That’s a huge discrepancy, and it implies that a sample selection based on CIP use introduces a massive bias. I’ve watched this poll grow over the weeks, half-expecting things to even out as the sample size increased. It didn’t. It has been pretty constant, with non-Ribbon non-CIP users outnumbering Ribbonite non-CIP users by a substantial margin.

Let’s put the groups together, shall we?

All voters (263)
Ribbon on 44% (116)
Ribbon off 56% (147)

So, if the voters in my poll were observed by Autodesk via CIP they would appear to be 69% Ribbon users. In fact, only 44% of these voters are Ribbon users.

How many AutoCAD 2010 users really have the Ribbon on? 69%? 30%? 44%? Some other number? I don’t know, and that’s not the point. The point is, Autodesk doesn’t know either. It can take some smart guesses, but just assuming CIP is accurate isn’t smart, it’s just a guess.

Why does this matter? Because Autodesk makes decisions based on this stuff. Decisions that affect you and me and how we use our tools. Have a look at this statement from Autodesk’s Teresa Anania, Director of Industry Management (taken from her interview with Deelip Menezes about Inventor):

…we had data that suggested that the new ribbon UI was well received and would be absolutely all that customers needed …. And now since we have the CIP data that shows us how our customers are using the software, we can analyze this before we permanently turn anything off.

Comments like this (and others from other Adeskers) seem to indicate that there is an unspoken assumption that CIP users accurately represent a true cross-section of users in general.

I know that Autodesk doesn’t rely solely on CIP; it uses a wide range of research tools to find out what users are up to and what they need. I regularly encourage you to participate in various Autodesk surveys, for example. But there are problems of accuracy inherent in all those methods. It would be natural, when faced with a set of apparently conflicting results from different sources, for Autodesk decision-makers to simply assume that the source with the biggest sample size is the most accurate. That could be a dangerous mistake, for both Autodesk and its customers.

Note: my arithmetic was off in several places when I posted this, and I have edited the post to correct some of the figures. These corrections do not invalidate the arguments; the substantial bias is still evident.