Kou, Yubo, and Colin M. Gray. “Supporting Distributed Critique through Interpretation and Sense-Making in an Online Creative Community.” Proceedings of the ACM on Human-Computer Interaction, Dec. 2017, doi:10.1145/3134695.
Yubo Kou and Colin M. Gray, Purdue University
May 17, 2020
Introduction
- This paper will help me understand how online critique is done on public platform
- Critique is defined as: “a dialogue in which the interjection of a reasoned opinion about a product or action triggers further reflection on or changes to the artifact being designed”
- Critique is a key process impacting designers to “extend their domain knowledge” by learning new methods
- Traditional methods of critique are in-person in studio settings
- Critique builds social relationships and can be spontaneous; physical location is important
- Distributed critique (DC):
- “a set of critique practices whereby geographically distributed creators engage in the critique of design artifacts and processes”
- Objective of this research:
- DC supports creativity and “diversity of economic entry points”
- DC is most effective form of critique for online creative communities; but isn’t it the only?
- There is a gap in academic studies for DC to identify issues related to creative work
- Mixed methods used
- Online creative community; Reddit; “r/userexperience/” subreddit
- Research questions:
- What design artifacts do community members offer for critique?
- How does distributed critique content differ from studio critique?;
- How do discourse practices enable critique and advice-giving to occur in this online community?
Lit Review: Studio Critique and its Typology
- Critique is part of studio pedagogy
- Where one develops professional communication, design discourse, refining process
- Four categories of critique:
- Formal: high stake evaluation with students, professors, jury members
- Seminar/group: classroom environment led by students/instructor
- Desk: 1-on-1 between student and instructor
- Peer: informal, context-specific interactions among students
- Genres of feedback (based on Dannels and Martin typology):
- Judgement
- Process-oriented
- Brainstorming
- Interpretation
- Direct Recommendation
- Investigation
- Free Association
- Comparison
- Identity Invoking
Lit Review: Crowdsourcing in Design Feedback
- Main difference between crowdsource feedback and studio critique is the structuredness
- Organizing online crowdsource feedback into categories is valuable
- High-quality critiquing criteria produces high quality expert feedback
- Crowdsourcing feedback draws non-expert feedback, which are not beneficial for complex interdisciplinary practices such as architecture and UX design
- Crowdsource feedback is used more often for visual design
- I may not find online CoPs for UX design because there is no structure to ‘weed’ out expert and non-expert feedback
- Reasons why UX design does not benefit from crowdsource feedback (non-expert feedback)
- “UX design includes a more complex temporal and dynamic array of elements… contexts, and user interactions”
- “UX is an emerging, multi-disciplinary field with no coherent body of knowledge”
- Sharing UX work can vary in the various process stages
- No “gold standard” for critiquing UX artifacts
- Do you have a gold standard on how to assess UX work? Is there a critique process you follow? Ex. Figma has published their critique process their product facilitates
- To critique a UX design artifact, one must understand the complex context it is intended for
Lit Review: Distributed Critique in Online Creative Community
- Design critique traditionally done in studios is also being done online; “this represents a departure from historic research on studio pedagogy where design work is produced, critiqued, and improved primarily through formal educational training”
- Distributed critique is a social practice; benefits go beyond just improving work
- Unique benefits to online critique community (that studios do not offer):
- Professional networking
- Improve specific aspects of work
- Gain “overall sense of the community’s perception”
Lit Review: Online Creative Community Focus
- Reddit = subject of online creative community
- Substantial user base
- Suitable environment to analyze distributed critique activity; it is a general-purpose creative community, not one specifically to moderate design critiques
- Methods; overview:
- Key word searches
- Snowball sampling to find subreddits
- 14 subreddits
- Descriptive statistics
- Chose ‘/r/userexperience/ subreddit; had most activity compared to other UX subreddits
- 8 months of data collection
- Important to note: Reddit is not explicitly for design critique
Method: Data Collection
- PHP, Reddit API, to collect post and comment data
- MySQL database for analysis
- API categorizes posts (new, hot, controversial, rising, top)
- 970 posts, 6958 comments
- Jan. 19, 2016 to Aug. 24, 2016
- Metadata was captured; title, author, timestamp, content, shared URL, number of comments, score
- Score = number of upvotes minus downvotes
Method: Data Analysis
- Researchers; 2 coders
- Content analysis to divide between sharing work and soliciting critique
- 90 critique seeking threads were analyzed
- Omitted 4 unique threads led by moderators; all subreddit members were invited to share portfolio for critique; call out for all members to share their work in a thread rather than talking about 1 user’s work on a thread
- Answering research question 1: What design artifacts do community members offer for critique?
- The two coders read all threads and shared initial thoughts on emerging themes
- Found that process UX design work is often shared; open coding was done to categorize artifacts by stage of project
- 4 stages were identified with distinct characteristics
- Codebook was created to document this process
- The two coders read all threads and shared initial thoughts on emerging themes
- Answering research question 2: How does distributed critique content differ from studio critique?
- “thematic analysis of the 569 comments using Dannels and Martin’s typology of critique”
- All comments were coded by typology; discussed disagreements
- Since I’m doing this alone, how do I discuss for disagreements?
- Answering research question 3: How do discourse practices enable critique and advice-giving?
- Discourse analysis
Findings: What Design Artifacts are Presented for Critique?
- Not limited to final design artifacts; work spread across entire design lifecycle
- Interesting note: “Notably, designers often talked about their self-perceived creative skill while requesting critique.”
- Four lifecycle stages identified:
- Preparation, ideation, work in progress, final work
- Preparation
- Presenters admit little to no UX knowledge in their questions and ask for general tips and recommendations on UX knowledge, tools, methods
- Ex. “Do you need separate designs for every Android phone on the market?”
- Ex. “When is the ‘best’ time to update your UX?”
- Ideation
- Covers mainly 2 situations: (1) ideas/inspiration for specific projects, (2) students with desire to launch career in UX and need advice on how to start
- Shows understanding of UX and asks for feedback on certain idea
- Ex. “Ecommerce Inspiration help :)”
- Ex. “I’m currently searching for a topic for my bachelor thesis. What do you guys [think] would be interesting to write about?”
- Work in progress
- Professionals with questions on specific feature or element of an in-progress artifact
- Seeks industry expert advice
- Ex. “Why did (ecommerce) registration rate go down at checkout when we tested social login buttons?”
- Final work
- Request for general feedback on final work
Findings: What is the Content of Distributed Critique?
- 569 comments were coded for genres of critique based on Dannels and Martin typology
- Frequency of genre type is shown in percentages
- Dannels and Martin typology is used as a framework for analysis
- Distribution of critique genre is “strikingly different” from studio critiques
- “In this regard, our findings about the strong presence of interpretation indicates that the online community supported a more equal, collaborative, and interactive critique environment that allowed design presenters to develop their own thoughts about their design, compared to studios where power disparity between student designers and critics are often present”
- Design presenters and critics had to collaborate more to interpret each other’s work and feedback since they were unfamiliar with each other’s expertise and knowledge
- This bar chart shows patterns in content analysis by critique typology and project lifecycle
- “The strong presence of interpretation across all stages indicates that the online community enabled both design presenters and critics to actively interpret the presented work.”
- For each pattern, researchers provide rationale based on content analysis (ex. students posting for earlier stages vs. professionals posting for latter stages)
- Not all critique typologies shows patterns
- Critique mostly occurred in the first responses to the thread and conversation did not continue further; unlike studio where the critique conversation is ongoing
- Nature of asynchronous communication; less follow-up
- Rich advice leads to less replies
- Interpretation and investigation critiques were most common in the earlier replies; to understand the artifact
- Effort to understand the artifact in distributed critique can be a barrier to feedback
- Sensemaking is required to produce meaningful feedback
Findings: How Did Discourse Practices Enable Critique?
- Focused on how one critique from one person influenced critique from another person
- Discourse analysis was done by examining the type of dialogue between creator and critic
- Ex. investigation > interpretation, interpretation > interpretation, direct recommendation > interpretation process oriented > interpretation
- The critic often asks questions on what the presenter means by x and y
- As presenter and critic have dialogue, their responses can include multiple critique typologies; and interpretation is most common because they rephrase what each other have said to check they are understanding it right
Discussion
- Reddit community invites members of diverse background and experience levels
- Users are unafraid to admit inexperience
- Distributed critique (DC) vs. studio critique (SC)
- In SC, members are already familiar with each other’s backgrounds and project goal; they can get into sharing expert feedback quickly
- In DC, members share the context of the work when asked (which incites interpretation discourse)
- Critique requests were freeform and unstructured since the sub-reddit is not a formal critique platform
- “interpretation and sensemaking were core activities for both design presenters and critics.”
- Collective sensemaking dominates discourse in distributed online communities; social process
Discussion: Distributed Critique as Dialogue
- “A conversation was able to be sustained because both sides not only contributed new content, but also showed passion and engagement for the topic.”
- In order to produce effective feedback, both parties had to actively participate in critiquing (clarifying, interpreting) each other’s feedback; this conversational “detour enriches the quality of feedback
- Each response in the thread was deeply embedded in the context of the conversation between poster and critic; not standalone comments
- Deep conversations are still achievable in distributed online communities
Discussion: Distributed Critique as Social Learning
- Distribute critique becomes a site for social learning
- Designers lean on other designers for knowledge because they do not have a formal source of knowledge to turn to
Discussion: Designing Socio-technical Systems to Support Distributed Critique
- “Perhaps paradoxically, the critique practices that emerged in this online subreddit community exhibited more similarity with physical critique environments than other systems that had been purpose-built to support specific types of critique practices”
- Platforms intentionally designed to facilitate design critique may be “too transactional in nature” and miss opportunities for fostering organic forms of dialogue