API Documentation Survey Results, 2010
In a perfect world, API documentation would contain clear, complete instructions on
everything that developers need to know to use your platform. In reality, organizations
have limited time and budget to create API documentation, and so organizations need
to prioritize to create documentation that is most useful to the people who will use it.
SDK Bridge set out to find out what is most important to the people who use APIs by sending
out a survey and asking them. We found, not surprisingly, that many thought that documentation
could be better. When asked to rate quantitatively, people rated overviews, sample code, and
API references the highest, followed closely by tutorials. When asked for open-ended comments,
a large percentage mentioned working sample code as helpful, and an equally high percentage
mentioned how important it is that documentation be complete and accurate.
Two other important factors that were mentioned by several respondents are
(1) it is very helpful to explain why you would use something in addition to how you use it and (2)
good overviews can really help when getting started with a technology.
Thirty six people respond to the survey. Half of them were either developers or architects, and other half were split
between being project managers, managers, writers, and testers.
When asked what kinds of APIs they were working with, the largest group said Web APIs, which is an indication
of just how fast the Web API industry is growing. Server and desktop SDKs were next, followed by a significantly-sized
group working with mobile SDKs, and a small number working with SDKs for gaming consoles.
Current State of Documentation
We asked people to rate the documentation that they are using. The answer came back
pretty clearly as "fair", with a few people responding "good" or "bad". No one rated documentation
as either "excellent" or "unusable".
Rating Types of Documentation
Participants were asked to rate the usefulness of overviews and conceptual material, API reference material, sample code,
tutorials, blogs, and forums. The following chart shows that overviews, API reference material, and sample code were
all rated the most useful, followed by tutorials, and then followed by blogs and forums.
What's interesting about this data is how it breaks down into content that is written by
professional programmer/writers as compared to content that is contributed by the developer community.
Overviews and reference materials (almost always written by professional programmer/writers)
are rated as between somewhat useful and very useful,
as is sample code (sometimes written by professional programmer/writers and sometimes contributed by the developer
community). Tutorials (also sometimes written by professional programmer/writers and sometimes contributed by
the developer community) are rated lower as somewhat useful. Forums and blogs (most often contributed by
the developer community) are rated lowest, between somewhat useful and not very useful. So the general
trend suggests that developers on average feel that content generated by professional writers is more
useful than that contributed by the developer community.
We broke down the data by roles to see if this had an effect on how these types were rated, but the
effect was minor, as shown in the chart below. It appears that testers tend to find documentation
less useful on the whole, with the exception of tutorials, which is not surprising.
In addition, we broke down the data by types of APIs to see if this had an effect on how these
documentation types were rated, but again, the effect was minor.
What Makes Documentation Helpful or Not Helpful?
Participants were asked what makes documentation helpful or not helpful.
Some common themes emerged.
Sample Code (30% mentioned)
30% of the respondents mentioned sample code as important, especially working sample code.
"The most useful API documentation I've seen shows the API calls with a working
example to show the response and let me tweak it," and "Examples are extremely helpful,
especially working samples. There's no substitute for actually seeing something work."
Examples should be clear and short, and they should "align to real-life expected use cases,
not be contrived to simply show the syntax of what it looks like to call a function
in a particular language."
Completeness and Accuracy (30% mentioned)
Completeness and accuracy were themes in 30% of the responses.
"It needs to be comprehensive, i.e., all the information has to be there.
You need to be able to find the answers to specific questions when you have them."
Also: "The vast majority of API documentation I have seen does not completely
describe the behavior and omits important details." Mentioned as often missing are:
- The range of acceptable arguments.
- Failure scenarios as well as success scenarios
- Examples of query parameters, not just descriptions
Usage (18% mentioned)
18% of the respondents mentioned that explaining how something
works is not enough, and that you need to make sure that documentation explains
why you would use it. "Providing examples of how the API can be used would be very helpful."
There appears to be a desire for usage scenarios for both the entire platform and
for specific calls. One person summed it up this way: "Focus on the 'why'."
Getting Started with a Good Overview (15% mentioned)
Several people mentioned that the documentation requires a good overview that
helps people get started. "It has to be presented in a logical order for beginners
who start out not knowing anything." Also: "Diving into details without an
overview is not helpful." Developers want to be able to start at a high level, but
be able to easily get to the low level details. One person wanted a "clear, concise
overview with drilldown to specifics and error codes," and another said the
documentation should be "presenting the complete picture across a technology
such that a new user can drop in at the level they need: overview down
to detailed reference."
Other Helpful or Unhelpful Aspects
Other useful suggestions include:
- Clarity is important: explanations should be in plain language.
- Avoid minimal documentation: don't explain a method or property by
using the same words that make up the method or property name.
- "Documentation is secondary to a good API. Without a good API the
documentation can only carry you so far."
- Use cross-referencing: "If you mention another API or concept, link to it so that
I can explore it directly."
- Images are important for GUI-related APIs.
- Documentation should be organized so that it's easy to find what you are looking for.
- Sample code should not do too many things unrelated to the API.
Participants were asked for any other comments about API documentation. Some responses
- "Good API documentation has a sense of being written by a human being who does
development on that platform."
- "Keep it light up-front. Too much data immediately tends to drown the documentation and alienate the user."
- "The best API documentation allows others to submit samples and tutorials for others to read."
- "If there isn't a short snippet on how to use an API component then there is likely a need to
extend the API to encapsulate the long snippet into something short."
- "A good example [of API Documentation] is the
The following conclusions can be drawn from the quantitative portion of the survey:
- On average, people consider the current state of the documentation they use to be "fair".
- The most useful types of documentation are: overviews, references, and sample code.
- Tutorials are also considered somewhat useful, but blogs and forums are considered less useful.
- There is no strong trend in the data due to either the role someone plays or the type of API they are using.
From the text-based part of the survey, we can conclude that
sample code and completeness are clearly very important, followed by usage information
so people understand why an API would be used and help getting started with good
overview material. Sample code should actually be working code so that developers
can start with it and build on it.
Completeness means more than just descriptions of the various
pieces of a method, property, or request; it means also describing the behavior, and
especially the error conditions. Accuracy is also important, which is why the development team
needs to set aside enough time for good technical reviews, something that can be
difficult to accomplish.
If your budget is limited, it appears that the best place to put effort into is to create good, working sample
code and making sure that your documentation covers everything that it should and that it is as
accurate as possible. You can increase the adoption of your platform by including real-world usage examples
and by making it easy for developers to get started through good overview material.