Our current gatsbyjs.org homepage is getting a bit long in the tooth — our incredible designer @fk built it out a year and a half ago, but a lot’s changed and we’re getting ready to update it.
Now, it was time to move to the next phase: user testing.
- 6 testers
- All of them know React
- 5 of them had never heard of Gatsby before
Back in November, we watched 8 people who had never heard of Gatsby before as they landed on the homepage and asked them “what is Gatsby” and “how do you decide if you want to use it?” The results were…not as accurate as we hoped!
Two problems became clear as a result:
- The first impression people have about Gatsby from above-the-fold and just below the fold on homepage is not accurate. Sources of more accurate understanding: Github READme file, diagram on homepage
We had not written down these key facts about the .org site and therefore sometimes struggled to stay aligned:
- Who is the audience for .org?
- What are our value propositions for that audience?
- What is the purpose of the homepage?
- How do we know if we’re succeeding in that purpose?
It is important to the Gatsby community to solve these problems, because tons of new visitors come to the gatsbyjs.org homepage each month: 130,000 in January alone! We want to make sure we’re accurately representing Gatsby’s value there and helping more developers experience Gatsby’s value by clicking the “get started” button.
How can we support first-time visitors so they experience Gatsby’s value powerfully and quickly?
Measuring that goal:
How many people click on “Get started” and then stay on the site? Are our monthly active users increasing?
We held a workshop that went like this.
We gathered a couple engineers, a product manager, and a product designer and assigned a couple special roles:
- Timer: someone to use a timer to make sure people move through the activities below.
- Note taker: at the end, someone needs to take notes during the “final feedback” section.
We briefly reminded ourselves of what the current homepage looks like on mobile, and then reviewed the following information on the requirements for redesigning the homepage:
- Optimize for the purpose of the homepage: to help as many developers as possible use Gatsby
- How do we know if we’re succeeding in that purpose? Increase in number of clicks on Get Started button and monthly active users growing
- What should the homepage communicate about Gatsby? “Gatsby is a free, open-source, modern website framework based on React to create and deploy blazing fast websites or apps with ease.”
- Who is the audience? developers who build sites for clients, companies, and their own projects
- What are our value propositions for that audience? fast sites, performance out of the box, productive from the start
We did 3 rounds of the following pattern and saw that in round two, our sketches diverged and in round three, they converged:
- 4 minutes: Everyone draw (with pen/pencil & paper) what they think the mobile version of the .org homepage ought to look like.
- 1.5 minutes: Upload picture to this Slack channel: #p-dot-org-prototypes
- 1 minute each: The person who drew the sketch being talked about can explain it for 1 minute, focusing only on facts. No pitching your ideas or persuading people.
- 2 minutes per person in which they share comments in these categories:
- Factual observations about this exercise.
- What I liked about these sketches and why.
- What I don’t like about these sketches and why.
- What I don’t understand.
We coalesced around: The homepage should be ordered by explaining the following:
- What is Gatsby?
- Why is it worth paying attention to?
- How do you use it?
We left the prototype workshop with a few concerns that could be validated through testing:
- Logos of companies could turn off developers b/c it’s cliche and sometimes meaningless or they could provide social proof that world-class engineers use Gatsby. We’ll test this. Possibility is to combine the logos with case studies so they aren’t in the classic “logos floating on white space” design that most startups use.
- Debate over how to help developers know that Gatsby also has some upcoming commercial offerings. What do we call the commercial offerings so they make sense to first time visitors?
“Install Gatsby” is an more specific call to action, so better than “Get Started” probably.
Spoiler alert! The button label “Install Gatsby” did not work in the usability tests. Nearly everyone thought that when they clicked on it, they would be taken to their phone’s app store.
Here is the final lo-fi prototype that we’ll use to guide the next phase of visual design. Below, I list the reasons we’re moving forward with this prototype.
The definition was clear in the prototype and the testers could explain what Gatsby is after glancing at the prototype for only a few seconds. This was a success!
We learned that Lighthouse scores weren’t as persuasive to testers as we anticipated because many people don’t know how the scores are calculated. Other ideas that could show Gatsby’s value more concretely might be case studies, sites in the site showcase, a diagram of how Gatsby works, code snippets, and more.
This is a question that testers had and the prototype didn’t quite answer it, so we concluded that we will simply answer that question above the fold on the homepage and then include a button labelled “Learn more” that leads to a list of the kinds of sites people build with Gatsby and resources for people who want to build each kind of site.
The “Install Gatsby” button label did not work in the usability tests. Nearly everyone thought that when they clicked on it, they would be taken to their phone’s app store. For this reason, we are sticking with “Get started” as a button label (this worked in usability tests).
We made two changes to fix the initial problems uncovered by usability testing. The “How it Works” diagram was moved up so some is visible above the fold, since that diagram was easier to quickly understand than the text that now is below the diagram. Also, the subheading under “How it Works” is changed to make it clear that Gatsby is open source and free, and make it clear that it’s a tool for developers.
Then we refined those changes by modifying the headline and subheadline.
We are also working on getting this RFC merged to work more on the homepage. We hope you comment on it!