In a perfect world, API documentation would contain clear, complete instructions on everything that developers need to know to use your platform. In reality, organizations have limited time and budget to create API documentation, and so organizations need to prioritize to create documentation that is most useful to the people who will use it.

SDK Bridge set out to find out what is most important to the people who use APIs by sending out a survey and asking them. We found, not surprisingly, that many thought that documentation could be better. When asked to rate quantitatively, people rated overviews, sample code, and API references the highest, followed closely by tutorials. When asked for open-ended comments, a large percentage mentioned working sample code as helpful, and an equally high percentage mentioned how important it is that documentation be complete and accurate. Two other important factors that were mentioned by several respondents are (1) it is very helpful to explain why you would use something in addition to how you use it and (2) good overviews can really help when getting started with a technology.

Who Responded

Thirty six people respond to the survey. Half of them were either developers or architects, and other half were split between being project managers, managers, writers, and testers.

When asked what kinds of APIs they were working with, the largest group said Web APIs, which is an indication of just how fast the Web API industry is growing. Server and desktop SDKs were next, followed by a significantly-sized group working with mobile SDKs, and a small number working with SDKs for gaming consoles.

Current State of Documentation

We asked people to rate the documentation that they are using. The answer came back pretty clearly as “fair”, with a few people responding “good” or “bad”. No one rated documentation as either “excellent” or “unusable”.

Rating Types of Documentation

Participants were asked to rate the usefulness of overviews and conceptual material, API reference material, sample code, tutorials, blogs, and forums. The following chart shows that overviews, API reference material, and sample code were all rated the most useful, followed by tutorials, and then followed by blogs and forums.

What’s interesting about this data is how it breaks down into content that is written by professional programmer/writers as compared to content that is contributed by the developer community. Overviews and reference materials (almost always written by professional programmer/writers) are rated as between somewhat useful and very useful, as is sample code (sometimes written by professional programmer/writers and sometimes contributed by the developer community). Tutorials (also sometimes written by professional programmer/writers and sometimes contributed by the developer community) are rated lower as somewhat useful. Forums and blogs (most often contributed by the developer community) are rated lowest, between somewhat useful and not very useful. So the general trend suggests that developers on average feel that content generated by professional writers is more useful than that contributed by the developer community.

We broke down the data by roles to see if this had an effect on how these types were rated, but the effect was minor, as shown in the chart below. It appears that testers tend to find documentation less useful on the whole, with the exception of tutorials, which is not surprising.

In addition, we broke down the data by types of APIs to see if this had an effect on how these documentation types were rated, but again, the effect was minor.
What Makes Documentation Helpful or Not Helpful?
Participants were asked what makes documentation helpful or not helpful. Some common themes emerged.

Sample Code (30% mentioned)
30% of the respondents mentioned sample code as important, especially working sample code. “The most useful API documentation I’ve seen shows the API calls with a working example to show the response and let me tweak it,” and “Examples are extremely helpful, especially working samples. There’s no substitute for actually seeing something work.” Examples should be clear and short, and they should “align to real-life expected use cases, not be contrived to simply show the syntax of what it looks like to call a function in a particular language.”

Completeness and Accuracy (30% mentioned)
Completeness and accuracy were themes in 30% of the responses. “It needs to be comprehensive, i.e., all the information has to be there. You need to be able to find the answers to specific questions when you have them.” Also: “The vast majority of API documentation I have seen does not completely describe the behavior and omits important details.” Mentioned as often missing are:

  • The range of acceptable arguments.
  • Failure scenarios as well as success scenarios
  • Examples of query parameters, not just descriptions

Usage (18% mentioned)
18% of the respondents mentioned that explaining how something works is not enough, and that you need to make sure that documentation explains why you would use it. “Providing examples of how the API can be used would be very helpful.” There appears to be a desire for usage scenarios for both the entire platform and for specific calls. One person summed it up this way: “Focus on the ‘why’.”

Getting Started with a Good Overview (15% mentioned)
Several people mentioned that the documentation requires a good overview that helps people get started. “It has to be presented in a logical order for beginners who start out not knowing anything.” Also: “Diving into details without an overview is not helpful.” Developers want to be able to start at a high level, but be able to easily get to the low level details. One person wanted a “clear, concise overview with drilldown to specifics and error codes,” and another said the documentation should be “presenting the complete picture across a technology such that a new user can drop in at the level they need: overview down to detailed reference.”

Other Helpful or Unhelpful Aspects
Other useful suggestions include:

  • Clarity is important: explanations should be in plain language.
  • Avoid minimal documentation: don’t explain a method or property by using the same words that make up the method or property name.
  • “Documentation is secondary to a good API. Without a good API the documentation can only carry you so far.”
  • Use cross-referencing: “If you mention another API or concept, link to it so that I can explore it directly.”
  • Images are important for GUI-related APIs.
  • Documentation should be organized so that it’s easy to find what you are looking for.
  • Sample code should not do too many things unrelated to the API.

Open Comments

Participants were asked for any other comments about API documentation. Some responses include:

  • “Good API documentation has a sense of being written by a human being who does development on that platform.”
  • “Keep it light up-front. Too much data immediately tends to drown the documentation and alienate the user.”
  • “The best API documentation allows others to submit samples and tutorials for others to read.”
  • “If there isn’t a short snippet on how to use an API component then there is likely a need to extend the API to encapsulate the long snippet into something short.”
  • “A good example [of API Documentation] is the Flickr API.

Conclusions

The following conclusions can be drawn from the quantitative portion of the survey:

  • On average, people consider the current state of the documentation they use to be “fair”.
  • The most useful types of documentation are: overviews, references, and sample code.
  • Tutorials are also considered somewhat useful, but blogs and forums are considered less useful.
  • There is no strong trend in the data due to either the role someone plays or the type of API they are using.

From the text-based part of the survey, we can conclude that sample code and completeness are clearly very important, followed by usage information so people understand why an API would be used and help getting started with good overview material. Sample code should actually be working code so that developers can start with it and build on it. Completeness means more than just descriptions of the various pieces of a method, property, or request; it means also describing the behavior, and especially the error conditions. Accuracy is also important, which is why the development team needs to set aside enough time for good technical reviews, something that can be difficult to accomplish.

If your budget is limited, it appears that the best place to put effort into is to create good, working sample code and making sure that your documentation covers everything that it should and that it is as accurate as possible. You can increase the adoption of your platform by including real-world usage examples and by making it easy for developers to get started through good overview material.