Quality of E-learning Systems Reports and Reviews
How Good Are Brandon-Hall's Reports at Evaluating LMS, LCMS CMS?
by Luigi Canali De Rossi
Recently I have received many requests for further information relating to the evaluation and selection of an appropriate LMS (Learning Management System) or LCMS (Learning Content Management System).
There are not indeed many ways to easily test, compare and evaluate different e-learning systems, features, functionalities and costs. The industry is almost oversaturated with over 1,000 vendors now competing for this growingly interesting market.
In most cases, to test or try any LMS/LCMS system one must contact directly one of the vendors and then only be allowed to be invited to a private demo, showcase or other "controlled" demonstration session.
It is indeed very hard to find vendors, like eCollege.com, that allow nowadays for a full test, download, or even hosted try out of their tools. Reasons and motivations for this vary, but the end result is that one has to be very prudent and sceptical about any comparative report, research dossier or other guide which is based on the above situation and which has not truly put to a real test the systems that are being evaluated.
The problems connected with this status quo are relative to the effective usefulness and credibility of the reports being compiled. In my personal experience, when e-learning vendors demonstrate and show their products, they demonstrate what the product can do, following a very specific pre-planned trail. I have seen demonstrators at all levels becoming less confident when a barrage of questions relating to your specific implementation are asked, and a frequent reference to the need for specific customization required or to features that will be coming in the next release.
Problem is we are really doing the beta testing, product specification and market positioning for many of these products, while companies are not acknowledging the strategic relevance we have in this game, and the value of the feedback we are able to provide them.
What I advise is therefore the need to approach vendors with very specific and detailed RFPs (requests for proposals), which must contain in detail your project story and your specific technical needs and requirements. Secondly we must approach these vendors with a clear prototype/pilot project in mind, so that those that really have the means and will to show how well their solution fits our need, can do so by contributing entirely or in part to the realization of a working model of your final project. This approach must be certainly leveraged more by institutions and organizations who feel too inexpert or unprepared to jump onto the e-learning bandwagon immediately.
The proposition of collaboration and partnership in the creation of a working prototype is always a valuable opportunity to propose, especially when you can offer to your vendor the prestige of mentioning the name of your organization in their marketing and promotional materials. It also undoubtedly offers to them a rich opportunity for real testing and for adding and improving many of the aspects of their products. This by itself, is a feature that we as customers should not give up in any way. Our feedback, testing and trying is more valuable to these companies than anything else.
In more than one instance in the recent past, I have had the feeling that some of the players in the e-learning arena have actually leveraged customers ignorance and limited preparation to their full advantage. I have seen systems that have been basically developed for one unique, large customer implementation, becoming brands and attempting to sell the system to the larger e-learning market by gradually adding and integrating gradually what others would also need and require.
I cannot blame the business logic of this, but I do reflect about the ethical issues behind and about the fact that customers are being mislead not being told the whole story behind the chrome. This is why we end up with products costing above one hundred thousand dollars, and which also require a lot of customization to work in the way you need.
At those prices you can have a whole offshore Indian development and programming team working for you for several years and developing exactly your custom solution, without your organization becoming a guinea pig for the rest of the industry.
In light of all of the above, I would quietly express a word of caution to the ones of you who have already spent or are seriously considering to spend the consistent dollar amounts required for the purchase of the only comparative reports on these systems available today.
While these reports do provide a lot of information, the information is completely provided by the vendors, as are the presentations and showcases of their products features. The editors and reviewers utilized by Brandon Hall also have limited ways to actually verify and test the actual application of most of the features and characteristics provided by these vendors.
My suggestion then is to understand that these reports are really nicely and well laid out collections of vendor's specified information. Not an actual report that has verified and tested systems to see what they can or cannot do.
It is easy to proclaim that one system for example can output to different media, including print, CD-ROM, Web, Presentations and the like. But seeing the process required, steps, and actual output you are going to obtain in this process, sheds a totally different light on what vendors say and what actually fulfil your requirements and needs.
I have seen in fact high-end systems in which though it was possible to produce files that were compliant with those different media, it produced them in a way that required manual re-editing of each to make them usable.
To see the reports I am referring to please go to:
http://www.brandon-hall.com
http://www.brandon-hall.com/ publications.html
Prices for these unique reports ranges from USD $ 245 to a $ 995. A comparative report of "Learning Management Systems" costs USD $ 795.
The interesting part is actually how the information is prepared and collected for these reports, as it can be read inside the Methodology paper available at this very site.
http://www.brandonhall.com/public/ publications/LMS2002/methodology.pdf
Just read carefully how they reviewed and critiqued LMSs. Only online or in-person demonstrations, reviewed each other's reviews, list of features and descriptions are all validated by the vendors.
Brandon-Hall's staff have never put their hands on any product in a way that can allow them to see what really works and what doesn't.
Perhaps my perspective is narrow, but after many experiences with the vendors, I wouldn't trust a compilation of features that have been provided and demoed by the vendors themselves.
It's no rumour: Brandon Hall says it.
blog comments powered by Disqus