ISSUE 03 - Q3 2002

Results from a survey of web prototyping tools usage

In June 2002 GUUUI conducted a survey on web prototyping tools usage. The purpose of the survey was to find out what tools are used for prototyping, what requirements interaction designers have for their tools, and how happy they are with the tools they are using. This article presents the results of this survey.

Some key findings:

There is currently no distinct consensus on what is the best tool to use for web prototyping
HTML and diagramming tools are the most commonly type of tools used for web prototyping
Macromedia Dreamveawer and Microsoft Visio are the most widely used tools
Respondents feel that the tools they use fall short on some high priority criteria, such as laying out and making changes to pages
Few seem to be perfectly happy with the tools they are using though users of HTML tools are strikingly more content than users of diagramming tools

What tools are used for prototyping?

The survey confirmed what discussions on mailing lists such as CHI-WEB and SIG-IA have indicated: there is currently no agreement on what is the best type of tool to use for web prototyping. However, there is a majority of 28.3% using a visual or text based HTML tool as their primarily type of prototyping tool, closely followed by 24.5% who prefer a diagramming tool.

	"What is the primarily type of prototyping tool you use for prototyping?" (52 replies)
	Non-computerized tool such as pen and paper (paper prototyping), white boards, Post-It notes and overheads	7.5%
	Presentation software such as PowerPoint from Microsoft	9.4%
	Graphic design tool such as Illustrator or Photoshop from Adobe	18.9%
	Diagramming tool such as Visio or SmartDraw	24.5%
	Visual or text based HTML tool such as FrontPage, Dreamveawer or HomeSite	28.3%
	A software development tool such as Microsoft Visual Basic or Borland Delphi	1.9%
	A multimedia tool such as Director from Macromedia or Authorware from Macromedia	0.0%
	Other	9.4%

Responses to the question "What is the name of the prototyping tool you are using" showed that among the respondents using HTML tools, Dreamweaver is most widely used, while almost all respondents using a diagramming tool depend on Visio.

How happy are designers with the tools they are using?

Asking how happy interaction designers are with the tools they are using reveals that there is few shouts joy. While designers using a HTML tool tend to be pretty content, respondents using a diagramming tool are surprisingly dissatisfied.

"I'm perfectly happy with the prototyping tool I'm using"
	Strongly disagree	Mildly disagree	Neither agree nor disagree	Mildly agree	Strongly agree
All replies (52 replies)	7.5%	37.7%	15.1%	32.1%	7.5%
Using a visual or text based HTML tool (15 replies)	6.7%	20.0%	6.7%	60.0%	6.7%
Using a diagramming tool (13 replies)	7.7%	76.9%	7.7%	0.0%	7.7%
Using a graphic design tool (10 replies)	10.0%	20.0%	20.0%	50.0%	0.0%

What are the requirements?

In the survey, the respondents where asked what they require from a web prototyping tool, and to what extend the tools they are using live up to the very same requirements. The purpose of these questions was to compare the features of the prototyping tools with the designer's requirements.

The results showed that prototyping tools primarily have to support:

Quick and easy revision of pages
Quick and easy layout of pages
An easy way to demonstrate the prototype to clients and team members
Quick and easy site wide revisions

Less important is it that a web prototyping tool:

Is able to simulate advanced functionality (e.g. scrolling and selecting items from a drop-down list)
Is easy to learn
Produces good-looking prototypes

The answers about whether the tools that the respondents use live up to their requirements showed that the tools fall short on some high priority criteria. They primarily fail in supporting:

Quick and easy revision of pages
Quick and easy site wide revisions
Quick and easy layout of pages

This table compares the questions "Considering your requirements for a web prototyping tool, to what extend do you agree with the following statements?" (first row against the question) and "Considering your personal requirements for a web prototyping tool, to what extend does the tool you use live up to the following statements?" (second row) (52 replies)
	Strongly disagree	Mildly disagree	Neither agree nor disagree	Mildly agree	Strongly agree
Quick and easy to layout pages	0.0%	0.0%	3.8%	15.1%	81.1%
Quick and easy to layout pages	2.0%	9.8%	7.8%	39.2%	41.2%
Quick and easy to make changes to pages	0.0%	0.0%	0.0%	11.3%	88.7%
Quick and easy to make changes to pages	3.9%	19.6%	15.7%	35.3%	25.5%
Quick and easy to do site wide revisions to the prototype	0.0%	0.0%	3.8%	30.2%	66.0%
Quick and easy to do site wide revisions to the prototype	19.6%	33.3%	13.7%	13.7%	19.6%
Able to simulate basic functionality (e.g. link between pages)	0.0%	3.8%	20.8%	35.8%	39.6%
	21.6%	7.8%	13.7%	19.6%	37.3%
Able to simulate advanced functionality (e.g. scrolling and selecting items from a drop-down list)	1.9%	13.2%	30.2%	41.5%	13.2%
	32.7%	13.5%	7.7%	21.2%	25.0%
Easy to demonstrate the prototype to clients and team members	0.0%	0.0%	0.0%	32.1%	67.9%
	0.0%	13.5%	11.5%	21.2%	53.8%
Suitable for carrying out usability testing	0.0%	7.5%	20.8%	32.1%	39.6%
Suitable for carrying out usability testing	5.8%	17.3%	23.1%	32.7%	21.2%
Produce a good looking prototype	9.4%	9.4%	26.4%	39.6%	15.1%
Produce a good looking prototype	0.0%	13.5%	19.2%	34.6%	32.7%
Fit in with existing development tools	3.8%	21.2%	32.7%	26.9%	15.4%
Fit in with existing development tools	0.0%	9.6%	46.2%	26.9%	17.3%
Inexpensive	1.9%	7.5%	41.5%	30.2%	18.9%
Inexpensive	3.8%	13.5%	28.8%	28.8%	25.0%
Easy to learn	1.9%	11.3%	28.3%	30.2%	28.3%
Easy to learn	5.8%	25.0%	11.5%	44.2%	13.5%
Easy to use	0.0%	0.0%	11.3%	45.3%	43.4%
Easy to use	1.9%	11.5%	21.2%	40.4%	25.0%

To be able to compare the different tools' support of the respondent's requirements, I've compiled a table summarizing the ratings of the individual tools. Most interestingly, the table shows that:

HTML tools have a high overall score, with site wide revision being the major problem
Diagramming doesn't score high on any criteria, and is considered particularly poor at supporting site wide revisions and simulating advanced functionality
The respondents using a presentation tool find that their tools are very easy to use and suitable for presentation and setting up pages
Respondents using non-computerized tools feel hat the approach fall short when having to do site wide revisions

This table summarizes the replies to the single choice matrix question "Considering your personal requirements for a web prototyping tool, to what extend does the tool you use live up to the following statements?" (the numbers are calculated as "neither agree nor disagree" divided by two plus "Midly agree" plus "Strongly agree).
	HTML tools	Diagramming tools	Graphics tools	Presentation tools	Non-computerized tools
	(15 repl.)	(13 repl.)	(10 repl.)	(5 repl.)	(4 repl.)
It's quick and easy to layout pages	90.1%	69.3%	77.8%	100.0%	87.5%
It's quick and easy to make changes to pages	90.0%	57.8%	61.1%	80.0%	37.5%
It's quick and easy to do site wide revisions to the prototype	73.3%	15.4%	16.7%	60.0%	25.0%
It's able to simulate basic functionality	93.0%	53.9%	35.0%	80.0%	50.0%
It's able to simulate advanced functionality	93.3%	15.4%	35.0%	20.0%	37.5%
It's easy to demonstrate the prototype to clients and team members	83.4%	50.0%	95.0%	100.0%	100.0%
It's suitable for carrying out usability testing	76.7%	53.9%	50.0%	50.0%	87.5%
It produces good looking prototype	96.7%	57.8%	100%	60.0%	50.5%
It fits in with existing development tools	73.3%	57.7%	85.0%	60%	50%
It's inexpensive	60.0%	61.6%	65.0%	100.0%	87.5%
It's easy to learn	60.1%	46.2%	60.0%	100.0%	75.0%
It's easy to use	83.3%	50.0%	60.0%	100.0%	100.0%

Thanks to Jens Hørup and dubidu for sponsoring this survey.

Text: Henrik Olsen