Perceptions 2010: An International Survey of Library Automation

by Marshall Breeding. January 27, 2011

Introduction

In this time of tight budgets where libraries face difficult decisions regarding how to invest their technology resources, it’s helpful to have data regarding how libraries perceive the quality of their automation systems and the companies that support them. This report, based on survey responses from over two thousands libraries, aims to give some measure of how libraries perceive their current environment and probes at their inclinations for the future.

Some libraries may refer to the results of this survey as they formulate technology strategies or even consider specific products. I urge libraries not to base any decision solely on this report. While it reflects the responses of a large number of libraries using these products, I hope this survey serves more as an instrument to guide what questions that a library might bring up in their considerations and not to make premature conclusions based on subjecive responses. Especially for libraries with more complex needs, it’s unrealistic to expect satisfaction scores at the very top of the rankings. Large and complex libraries exercise all aspects of an automation system and at any given time may have outstanding issues that would naturally result in survey responses short of the highest marks.

The survey results also aim to provide useful information to the companies involved in the library automation industry. While each company likely performs its own measures of client satisfaction, this survey may show perceptions in different ways. I hope that the rankings in each category and the published comments provide useful information to help each of the companies hone in on problem areas and make any needed adjustments to their support procedures or product directions.

This year marks the fourth time that I’ve carried out this survey. Each year I have received around 2,000 responses, and a few more libraries responded this year than in any of the previous iterations. In very broad terms, the survey results are similar this year, but with some interesting new trends.

General Information about the Survey

This report describes the results of a survey that I conducted to gather data regarding the perceptions of libraries toward their automation systems, the organizations that provide support, and the quality of support they receive. It also aims to gauge interest in open source library automation systems.

I conducted a similar surveys in 2007, 2008, and 2009.

This year, I received 2,173 responses from libraries in 60 different countries. The countries most strongly represented include the United States (1,690 responses), United Kingdom (92), Canada (117), Australia (76) and New Zealand (44). As with the general demographics of the lib-web-cats database, the respondents of the library primarily come from libraries in English-speaking countries. Survey results were gathered between December 8, 2010 and January 19, 2011 (Full demographic summary).

The survey attracted more responses from libraries using Millennium (395), Symphony (282), Horizon (185), Voyager (124), OPALS (106), ALEPH 500 (106), Library.Solution(105), Polaris (104), Apollo (84), VERSO (72). There were fewer than 70 responses for each of the other ILS products represented in the survey. Systems with less than 20 did not appear in the main statistical tables. These responses can be seen through the individual ILS Product Reports available.

This article is an original publication of Library Technology Guides and is not slated to appear in any print publication. Please direct any comments or enquiries to the author.

This survey and its analysis reflect my ongoing interest in following trends in the library automation industry. It is designed to complement, and not replace, the annual Library Journal. The survey underlying the Library Journal article relies on information provided by the companies that offer library automation products and services. The survey that serves as the basis for this article collects data from the libraries themselves.

Survey Results

Statistics related to the question: How satisfied is the library with your current Integrated Library System (ILS)?

An interactive version of the statistical reports is available here, which includes the ability to view the responses for each of the ILS products, along with the redacted comments.

---

General Observations

This year top marks cannot be explained by the profile of library served, other than the general tendency for the companies that serve larger and more complex libraries fall into a middle or lower tier of rankings. The products earning top marks, Apollo, OPALS, and Koha (when supported by ByWater Solutions) differ in many ways. Biblionix offers Apollo as a hosted service and is proprietary and serves small public libraries; OPALS, is an open source ILS developed for K-12 school libraries and has been adopted by some church and synagogue libraries; ByWater Solutions provides support for the open source Koha ILS to mostly public libraries, through some academics responded as well. It's clear that within this top tier of rankings that the libraries responding perceive their automation system as well-suited to their needs, have received excellent support, and have a high regard for the company supplying the system.

Perceptions Open Source ILS Products and Support Companies

The scores in the “Interest Level in Open Source” naturally run high for those libraries already involved with an open source ILS, ranging from a perfect 9.0 given by those running Koha supported by ByWater Solutions, through 8.6 for Evergreen. For those running a proprietary ILS, the interest in open source ILS seems mostly indirectly proportional to satisfaction with the ILS, company, and support. Libraries running proprietary products that rate high satisfaction with the ILS, company, and support categories selected lower levels of interest in open source alternatives, while those more dissatisfied show at least some higher interest. The scale of interest in open source from those running proprietary systems tops out at 4.4 (Horizon) compared to scores greater than 8.26 from existing open source practitioners.

Tracking the performance of the Koha open source ILS is a bit more complicated since libraries work with any of a number of firms that provide support services and some operate it on their own, independent of any commercial organization. Some of the support firms represented in the survey include LibLime, PTFS, which now owns LibLime, PTFS Europe (independent, but partially owned by PTFS) ByWater Solutions, CALYX, Catalyst, Libriotech, and Prosentient Systems. Koha, when supported without the support of a commercial support firm (Koha – Independent) received the highest rankings for satisfaction with the ILS, but received somewhat lower scores for support and company satisfaction. In these cases, the libraries were essentially rating themselves. Koha when supported by ByWater Solutions received exemplary perception rankings for company satisfaction (8.41) and support (8.44). ByWater Solutions attracted the highest score in responses to the company loyalty (8.73) question.

OPALS, an open source ILS developed and is supported by MediaFlex used primarily in K-12 School libraries and in church and synagogue libraries received exceptionally high rankings in all of the survey categories. For many of the libraries responding to the survey, support and hosting is provided by the BOCES School Library Systems in the state of New York. Libraries using OPALS rated the customer support at the top of this category (8.76) and gave it second highest rankings in ILS satisfaction (8.43), company satisfaction (8.63), and company loyalty (8.71). The number of responses from OPALS libraries was dramatically higher this year (100) compared to last year (41), or 2008 (5 responses). The comments written by libraries using OPALS overwhelming lent praise to the product and its support.

Comments on the Products and Companies specializing in the Automation of Smaller Libraries

Apollo, a hosted ILS provided by Biblionix used by small public libraries, received top rankings for ILS satisfaction (8.58) and company satisfaction (8.72); second best for ILS support (8.75); and third for company loyalty (8.70). The number of responses for Apollo has increased over the four years of the survey: 81 in 2010, 35 in 2009, 7 in 2008, and 7 in 2007. None library using Apollo indicated consideration of moving to a new ILS, none plan for a discovery interface, and 78 out of 84 report on-schedule installation.

VERSO from Auto-Graphics received very positive ratings in each of the categories. In the first three years of the survey, ratings for VERSO showed gradual improvement; this year rating in each category dipped a notch. Perceptions for support ranked fourth (7.99), third for loyalty to company (7.96), fifth for company satisfaction (7.8), and seventh for ILS satisfaction (7.4). Libraries using VERSO ranked very low in interest to an open source ILS (2.24). Overall survey results confirm very high perceptions for VERSO as an ILS and Auto-Graphs as a company and comments offered reflect high praise and flag no significant problems.

Comments on the Products and Companies specializing in the Automation of Larger Libraries

Polaris attracted exceptional rankings from the 101 customer libraries that responded to the survey and the comments offered were overwhelmingly positive. In 2007, the first year of this survey, Polaris was the top performer in all categories. This year Polaris received almost exactly the same ratings for ILS satisfaction as it did in 2007, but placed fifth in the rankings. When looking at the four-year run of survey results, Polaris was rated very consistently in satisfaction perceptions each of the categories. Polaris rated fifth highest in ILS satisfaction and support satisfaction and company loyalty and placed fourth for company satisfaction. The survey results do not indicate any slippage in the positive perceptions of Polaris by the libraries that use its product; its lower placement in the ranks has more to do with the large numbers of responses from highly satisfied libraries of other products, mostly from the smaller-library arena.

Millennium from Innovative Interfaces, Inc. received quite respectable rankings, generally placing near the top of the middle tier of most categories. Libraries running Millennium responded to the survey in higher numbers than any other ILS (383). When considering the products that serve larger libraries or consortia, Millennium placed lower than Polaris in ILS satisfaction (7.11) and company satisfaction (6.66), but higher than products such as Evergreen, Symphony, Aleph, Voyager, or Horizon. Company loyalty rankings were slightly lower (6.40). The statistic that stands out the most for Millennium reflects the number of libraries indicating interest in migrating to a new ILS. This percentage has increased steadily since 2007 from 6.69% to 8.28% to 11.71% to 18.73% in 2010. The comments generally had a negative tone, many of which acknowledged the strengths of the system while complaining about the cost or lack of openness.

Libraries using Library.Solution from The Library Corporation responded with rankings that generally fall around the top of the middle tier. In the ILS Satisfaction category, the 7.09 mean rating fell just below Millennium; but in company satisfaction (7.23) and support satisfaction (7.32) scores ranked higher than Millennium. Support satisfaction for Library.Solution has improved each of the four years of the survey. Libraries gave average company loyalty ratings of 6.84 and 14.2% indicate interest in migrating to a new ILS.

SirsiDynix offers two ILS products represented in the survey, Symphony, (282 responses) and Horizon (185 responses). The two ILS products received remarkably similar scores. Though SirsiDynix promotes Symphony as its flagship ILS, it offers continued support and development for Horizon. In the last year, SirsiDynix implemented a major reconfiguration in its customer support operations, centralizing into its facility in Provo, UT and deemphasizing support in its international offices. The survey specifically asks respondents to base their rankings on the experiences of the last year. Given that the reconfiguration was made in [], this year’s perceptions in support may be at least partially due this new arrangement. Some comments from international sites did reflect concern. Support scores for Symphony have improved over the last three years 4.91 in 2008 to 5.33 in 2009 to 5.66 in 2010, though still relatively low relative to products from other companies. Just over 20 percent of the libraries with Symphony reported consideration of a new ILS. More than half of Horizon (105 out of 185) sites show interest in moving to a new ILS, though 25 of these libraries include Symphony among the replacement candidates. When comparing this survey to last year’s rankings, satisfaction for the Horizon ILS has dropped slightly from 6.07 to 5.99, company satisfaction increased from 4.91 to 5.31, while support satisfaction held steady (2009 = 5.77; 2010 = 5.73). For Horizon libraries, company loyalty ranks low at 4.94. Though still a wide mark below those libraries already involved with open source ILS products, libraries using Horizon indicated a higher probability of shifting to an open source ILS than libraries using any other proprietary ILS (4.44). 34 percent of Horizon libraries and 28 percent of Symphony libraries indicate interest in implementing a new discovery interface.

Two products from Ex Libris were represented in the survey results, Aleph, (106 responses) and Voyager, (124 responses); the company continues support, marketing, and developmlent for both of these ILS products. Aleph and Voyager serve large and diverse organizations and offer very complex functionality, placing them into a tier of products that do not receive superlative marks. Aleph received a mean score of 6.41 on ILS satisfaction, ranking it 15th; Voyager’s 6.01 put it two places below that. Perceptions of the company between these two products came out about the same: Aleph (5.97) and Voyager (5.90), as was customer support: Aleph (5.96) and Voyager 5.79); both showed similar company loyalty: Aleph (5.98) and Voyager (5.77). In the area of interest in open source, Voyager libraries ranked much higher (4.07) than those with Aleph (3.4). One of the dynamics of interest with libraries using either of Ex Libris’ ILS products involves any indicators of interest in the company’s upcoming Alma platform, positioned as an eventual transition path for Aleph and Voyager. The modest loyalty rankings do not reflect overwhelming interest at this early date. A number [how many] libraries that flagged interest in migrating to a new ILS in the near future mentioned Alma (or URM as it was called during the survey period).

---

Details about The Survey

The survey instrument included five numerical ratings, three yes/no responses, and two short response fields, and a text field for general comments. The numeric rating fields allow responses from 0 through 9. Each scale was labeled to indicate the meaning of the numeric selection.

Four of the numeric questions probe at the level of satisfaction with and loyalty to the company or organization that provides its current automation system:

• How satisfied is the library with your current Integrated Library System?
• How satisfied is the library overall with the company from which you purchased your current ILS?
• How satisfied is this library with this company’s customer support services?
• How likely is it that this library will purchase its next ILS from this company?

A yes/no question asks whether the library is considering migrating to a new ILS and a fill-in text field provides the opportunity to provide specific systems under consideration. Another yes/no question asks whether the automation system currently in use was installed on schedule.

Given the recent interest in new search interfaces, a yes/no question asks “Is the library currently considering a search interface for its collection that is separate from the ILS?” and a fill-in form to indicate products under consideration.

The survey includes two questions that aim to gauge interest in open source ILS, a numerical rating that asks “How likely is it that this library would consider implementing and open source ILS?” and a fill-in text field for indicating products under consideration.

The survey concludes with a text box inviting comments.

View the survey. (This version of the survey does not accept or record response data.)

In order to correlate the responses with particular automation systems and companies, the survey links to entries in the lib-web-cats directory of libraries. Each entry in lib-web-cats indicates the automation system currently in use as well as data on the type of library, location, collection size, and other factors that might be of potential interest. In order to fill out the survey, the responder had first to find their library in lib-web-cats and then press a button that launched the response form. Some potential respondents indicated that found this process complex.

The link between the lib-web-cats entry and the survey automatically populated fields for the library name and current automation system and provided access to other data elements about the library as needed. The report on survey response demographics, for example, relies on data from lib-web-cats.

A number of methods were used to solicit responses to the survey. E-mail messages were sent to library-oriented mailing lists such as WEB4LIB, PUBLIB, and NGC4LIB. Invitational messages were also sent to many lists for specific automation systems and companies. Where contact information was available in lib-web-cats, and automated script produced e-mail messages with a direct link to the survey response form for that library.

The survey attempted to limit responses to one per library. This restriction was imposed to attempt to sway the respondents to reflect the broad perceptions of their institution rather than their personal opinions.

The survey instrument was created using the same infrastructure as the Library Technology Guides web site—a custom interface written in perl using MySQL to store the data, with ODBC as the connection layer. Access to the raw responses is controlled through a user name and password available only to the author. Scripts were written to provide public access to the survey in a way that does not expose individual responses.

In order to provide access to the comments without violating the stated agreement not to attribute individual responses to any given institution or individual, an addition field was created for “edited comments.” This field was manually populated with text selected from the “comments” text provided by the respondent. Any information that might identify the individual or library was edited out, with an ellipse indicating the removed text. Comments that only explained a response or described the circumstances of the library were not transferred to the Edited Comments field.

Statistics

To analyze the results, a few scripts were written to summarize, analyze, and present the responses.

In order to avoid making generalizations based on inadequate sample sizes, the processing scripts included a threshold variable that would only present results when the number of responses exceeded the specified value. The threshold was set to a value of 20.

For each of the survey questions that involve a numeric rating, a set of subroutines was created to calculate and display simple statistics.

• Responses indicates the number of survey responses that made a selection for this question.
• A Response Distribution array lists the number of responses for each possible value from 0-9.
• The Mode indicates the numeric response that receive the most selections.
• The Mean is the average response, calculated by adding together all the responses and dividing by the Responses value, rounding to two significant decimal places.
• The Median is the middle response, calculated by placing each of the responses in a sorted array and selecting the middle value.
• The Standard Deviation was calculated by subtracting each response value from the mean, squaring the difference, summing the squares and dividing by the number of responses to determine the variance. The standard deviation is the square root of the variance.

The "survey-report-by-category.pl" script processes each of the numerical ratings, displaying each of the statistical components listed above for each product that received responses above the threshold value. This report provides a convenient way to compare the performance of each ILS product for the selected question. The report sorts the statistics for each product in descending order of the mean. The report categories available correspond to the survey questions with numerical scale responses.

The “survey-product-report.pl” script provides the results for each of the ILS products mentioned in the responses. This report also provides the statistical components for each of the numeric question. It also provides the percentage of yes responses to the two yes/no questions:

• Is this library currently considering migrating to a new ILS?
• Is this library currently considering acquiring a search interface for its collection that is separate from the ILS? The report then lists the “EditedComments” text.

[The text of this section mostly replicates what appeared in the 2007 version of this article. For for both editions of the survey I followed the same methodology for collection and and statistical analysis.]

---

Caveat

As I noted with previous editions of the survey, one should not read too much into the survey results. Responders to the survey provide their subjective impressions to fairly general questions. Although the survey instructions encourage responders to consider the broader institutional perceptions, it’s usually the case that multiple opinions prevail within any given library. While I believe that this survey does provide useful information about the experiences of libraries with their current integrated library systems and the companies that provide support, it should not be used as a definitive assessment tool.

Maintained by Marshall Breeding Copyright 1999-2023