The Beast that is API Documentation
Developers Developing for Developers
Some Background
In the summer of 2021, I embarked on a rotational project at Pluralsight to work with the internal Platform as a Service team. This team manages the experience powered by GraphQL that enables Pluralsight customers to access the API.
I was granted this opportunity for 3 months as part of my learning and career development in the area of Product Management.
My goal in working with this team was to assess the unknowns, identify the biggest areas of opportunity, and deliver high value to the customer. Much like Beast from X-men, APIs are complex and interesting. I fell in love with the beast, and maybe you will too after reading this article.
The Problem Brief
The developer.plurarlsight.com site was supposed to be self-service, however in its previous state, the page was proving to be difficult to navigate and could take users that aren’t familiar with GraphQL a while to get up and running.
Additionally, there was a visual disconnect between the static (getting started) pages and the dynamic (schema) pages.
There was no tracking or analytics on the developer API site to identify current customer journeys through the pages. This was based on the number of times an API was accessed, not necessarily the use of the developer portal. It was also difficult to identify how much time someone was spending on the site.
Identifying the Customer
My biggest challenge was defining who the customer was of the API. When I came into the team, the director had an idea of who the customers were based on past experience. But very little research had been done recently to validate that these personas were still up to snuff.
I quietly observed the team that was building out the APIs for the existing portal. The developers within the team used the portal themselves and other internal staff leveraged the platform.
This led to some bias around what the team thought they should be building. I frequently asked the question, “Why are we building this?” I heard statements like, “We have our own pain points when using the documentation, so I think we have a good handle on what needs to be done,” or, “Based on Professional Services feedback we are going in this direction.”
They had fallen into a trap and weren’t really sure why they were putting in time and energy into certain features. Don’t get me wrong, people in customer support and sales roles can be some of your greatest allies, but it’s all second hand anecdotal detail. It’s best to go straight to the source.
Customer Research
First I want to emphasize that customer research is not a linear exercise. It should be done regularly, and often. A job to be done that a customer has today could change tomorrow based on a variety of environmental factors and market trends.
However, with my rotational project, the customer research phase happened much later in the process. This was beyond my control as an intern was assigned to the project prior to my joining the team. The software engineer on the project jumped right into coding without any findings to back up the proposed solution. It was all based on team agreements.
I will also add that a product designer was not yet on the project, so design elements were rather rough. The UI for the API documentation and general navigation had not been evaluated since its launch in 2018.
Notice in the old navigation that the schema is at the top, and the quick start documentation is on the left navigation. The development team hypothesized that the schema could dynamically generate instead of having multiple lines of GraphQL to read through.
Customer Interviews
API documentation as a standard should be self service. If you type in “API for blank company” you’ll see what I’m talking about. So one of the outcomes we were hoping for was to make the site as user friendly as possible for any skill level with no personal guidance from a Pluralsight employee.
Over the course of a few weeks, I paired with a designer to dig into some of the customer behaviors when using the API. We pursued interviewing customers within our own organization that had little to no experience with GraphQL and that did not use the the developer portal landing page. These were considered our “newbies”.
I also pushed to talk to a couple of paying customers who actively engaged with the site. I had to get creative in finding these folks by parsing through customer tickets and talking to the technical support team. These were labeled “power users”.
Watching the User
Before an interview, you may have an assumption about a workflow that can be completely turned upside down simply by watching someone do their work. I love this part of the process.
In my research with the API I wanted to understand the navigation behavior and the types of tasks each user was trying to accomplish. We worked off of a script that had questions such as “What are you seeing?” and “What are you expecting here?” “Explain to me where you are going and why?” But the most important question was the rating. We wanted to gauge the usability of the site and asked the users to rank the usability from 1–5 with 1 being the worst and 5 being dream land.
It was interesting to see how a power user navigated the experience vs. someone who had never seen the interface before. One thing I learned from this exercise was that people create mental models for themselves and work around issues without really thinking about it. (The power user was actually more hesitant to adopt the simpler click path because they were so used to the old navigation!)
We asked the participants to rank how frequently they accessed the page and to compare it to other developer portals they have leveraged. Next, we gave the user a task to run a moderately advanced query. We gave no guidance, and instead watched the behavior of the developer attempting to complete the task.
What we discovered over the course of our research was the following:
Developers are busy. They don’t want to have re-learn something that they only do every few months. They needed an example query pre-populated for them so they could quickly identify how to query the data for their specific organization.
The order of the query/mutation was confusing. In fact, a lot of the walk through was confusing. So we added descriptions to the top of each schema doc.
Prior to release, we ran beta tests with some internal power users through professional services. We identified bugs and potential risks and then charged ahead!
This was a very fulfilling project during my learning journey as a product manager. I got my feet wet with customer research, was able learn from a product designer, and successfully launched a feature set! The beast that is APIs isn’t a horror film type of scary. But really is like Beast from X-Men. Very powerful and smart when used correctly. ;)
