In Design Study Methodology: Reflections from the Trenches to the Stacks, Sedlmair et al. (2012) use a reflective methodology to establish a much used framework for applied visualization research, which in itself generates knowledge through reflective practice.

This paper claims that "reflection is where research emerges from engineering", and provides this guidance for reporting on design study research: “a careful selection of decisions made, and their justification, is imperative for narrating a compelling story about a design study and are worth discussing as part of the reflections on lessons learned."

But, "a bit of retroactive introspection" is called out as being inadequate: "reflecting on lessons learned from the specific situation of study in order to derive new or refined general guidelines typically requires an iterative process of thinking and writing.”

What does reflection in the context of design studies actually involve? How does it work? How do we use it to generate and report on research findings in which we can have trust? How is it done well, and not so well? How does experience inform this? In short - what have we as a community of researchers and practitioners learned from the many design studies that have been undertaken? 

We think it’s time to reflect upon this.

A panel at our major conference provides an excellent opportunity to do so. We will bring together a group of researchers with different perspectives and backgrounds to define reflection, share experiences, and establish best practices for generating useful and trustworthy research results from design study research. In particular, this forum will provide a unique opportunity for cross-study reflection to:

• • • establish what has (and has not) been learned both through and about reflection in design studies;
    • identify commonalities and differences between design study experiences - when, where and how does reflection occur?;
    • explore ways in which reflection can contribute valid and transferable knowledge to the visualization domain;
    • discuss ways in which design study methodology might be redesigned or adapted to more effectively support cross-study visualization research;

In advance, we will ask a series of visualization researchers from academia and industry, with a range of backgrounds and experience in a different  application areas to answer five key questions in short position statements.

1. 1. 1. When and how do you reflect when conducting a design study?
      2. How do you record the process and results of any reflection during a design study?
      3. How might we make the contributions generated through reflection in design studies more trustworthy and useful?
      4. How and when does cross-study reflection occur?

• do you learn from or transfer knowledge to other design studies, beyond your own experience?

1. 1. 1. Has this reflection on your own practice resulted in any thoughts or ideas?  

Structured discussion will seek commonalities and differences in experience and suggestions for ways in which experiences gained through applied visualization work can be harnessed to establish research findings that are valid, trustworthy and usable.

We anticipate the panel informing the next generation of applied visualization research through reflection on reflection in design studies.