Human movement as a framework for developing computational representations of interactions

Deray, K

Human movement as a framework for developing computational representations of interactions

Deray, K

Permalink

Publication Type:: Thesis
Issue Date:: 2009

Closed Access

	Filename	Description	Size
	01front.pdf	contents and abstract	13.5 MB	Adobe PDF	View/Open
	02whole.pdf	thesis	178.55 MB	Adobe PDF	View/Open

Copyright Clearance Process

Recently Added
In Progress
Closed Access

This item is closed access and not available.

Full metadata record

Field	Value	Language
dc.contributor.author	Deray, K
dc.date.accessioned	2015-06-03T03:40:25Z
dc.date.available	2015-06-03T03:40:25Z
dc.date.issued	2009
dc.identifier.uri	http://hdl.handle.net/10453/35970
dc.description	University of Technology, Sydney. Faculty of Information Technology.	en_US
dc.description	NO FULL TEXT AVAILABLE. Access is restricted indefinitely. The hardcopy may be available for consultation at the UTS Library.
dc.description.abstract	NO FULL TEXT AVAILABLE. Access is restricted indefinitely. ----- Overview This thesis establishes a methodology for designing formal representations, visual languages, and visualisations of information about interactions. The conceptual modelling that enables the method for designing representations of interactions follows the elements of human movement. The representations of interaction process are modelled on bodily knowledge with the explicit objective of enhancing the semantics of the unfolding of interaction process. Interactions play a central role in our life. We interact with our colleagues and professional team members in the office, with our doctors during the visits to healthcare practices, with our lecturers and classmates in university. Through the interactions with the various computing systems that constitute our contemporary environment, we learn about it and achieve our goals. Interactions are situated at the very heart of the activities performed in many computerised domains, yet they remain poorly understood. Problem Formulation Each interaction constitutes a process that unfolds. This is the view the thesis takes. The way interactions unfold informs us about many aspects of the interactions themselves. Hence, in order to improve processes that rely on interactions, it is important to develop methods of encoding the way interactions unfold, capable of representing that information and allowing both humans and machines to utilise such information in order to improve interactions and, consequently, the processes that rely on them. For example, in health care, the overall process of patient treatment is a process that relies on interaction. It involves a series of interactions between patient and practitioner. If we consider such interaction in the context of the treatment process, then the overall treatment process will benefit. This research is about interaction, the interaction process, representation of such a process, and the manner in which bodily knowledge can be applied to the modelling of interactions. Interactions are complex, varied and dynamic. They are situated ‘transactions’ that derive meaning from the context within which they are embedded. The thesis addresses the problem of representing information about the interaction process that are machine usable. It seeks to develop a representation that allows capturing information about the unfolding of interactions. A central modelling issue is how to develop a representation of interactions that communicates meaning about the process without obscuring it. As it is undesirable to ‘obscure’ the representation of interaction the choice has been made to observe the act of interacting through the system of human movement. Human movement provides us with two important components for representing interactions; perception-action coupling and dynamics. This is suitable for the purposes because human movement by definition is a ‘reflective system.’ Through the output of this observation a representation of interaction can be defined. If the representation is modelled through the interacting processes that are integral to the representation of human movement interaction, then, by definition, the representation will partake of the same, or similar, characteristics as the system from which it was derived. There is then no a priori structure imposed on the construction of the interaction. Rather the phenomenon of interaction itself is the representation constructed. This methodology promotes a distancing of the work from the inherent subjectivity associated with content analysis approaches to representing interactions. Such models look to pre-defined selected features of interactions to provide definitions. In contrast, the approach in this thesis is to study the essential behaviour of interacting. Rather than starting with predefined notions of interaction, intuitive understandings of interaction based on experience form the initial basis. It is important to understand what interaction is, or what it is not, before any representation of interaction can be modelled. The general trend has been to consider interactions in terms of output, the results of the interaction, with little emphasis on the processes that construct interactions over time. There are problems with this approach as interactions are reduced to terms of output only and substantial part about them, the characteristics of the process of how they unfold, is not captured. Consequently the formalism developed in this thesis addresses this problem by capturing information about interactions that; - will improve the overall process, as it will allow us to extract knowledge about the way interactions unfold and incorporate that knowledge into the process; - will allow different views of that information to be accessed; - present, how such interactions unfold so that judgements can be made as to whether or not there has been a good communication. If provided with such profile of previous interactions in the beginning of a session, one can predict whether the current interaction unfolds well, and if not, based on the previous reflections, can respond with a strategy that can improve the interaction. The mentioned developments made in this thesis offer a range of contributions as summarised below. Thesis Contributions This thesis establishes a methodology for designing formal representations, visual languages, and information about visualisations of the interactions process. The conceptual modelling that enables the method for designing representations of interactions is inspired by Lakoff and Johnson’s approach to metaphors (Lakoff and Johnson, 1980). It allows expressing the target domain through the constructs of the source domain. In this thesis the source and target domains are human movement and interaction processes respectively. In the process the source domain is interpreted and formalised through well founded models of the system of human movement developed in movement observation science (Newlove & Dalby, 2004). From this are derived the constructs of elasticities and qualities. The behaviour of these two groups of constructs provides the shaping affinities for building expressions of interactions in the target domain. The mapping takes concepts that describe two frames of reference of human movement: (i) body position (the place of the body in space); and (ii) body dynamics (the motion that causes and expresses change from one position of the body to another), and uses their computational representations in the interactions domain. The set of constructs considered in this thesis include the following elasticities - the rising and sinking (RS-) elasticity and the contraction and extension (CE-) elasticity. The behaviour of elasticities describes the reciprocal effects between interacting parties and compactly expresses patterns of interaction. Below is a brief description of the thesis contributions. - Conceptual modelling of operating through the framework of metaphor between two physical systems, in particular human movement and interactions. The conceptual modelling establishes links between two physical systems by modelling one of them through the integral dimensions of the other. In the thesis, interactions as a system are modelled through the integral dimensions of human movement. - A methodology for modelling interactions. The methodology provides an analysis and transfer of the integral dimensions from one domain to another. Starting with observation of human movement, specifically over improvisational dance, it is then possible to derive concepts to model a formalism that behaves in a same manner. The methodology, called the InteractionSystem methodology, provides key insights of bodily knowledge; the way humans structure behaviour through bodily reasoning, association, and memory. From the observation of the phenomenon of interaction through human movement, the key constructs - elasticities and qualities - are derived. - A framework for interpreting interactions as a physical system. An interaction can be understood as a system through its components and the relations between them. This thesis has developed a multi-layer framework that provides key concepts of the system. - A language for expressing information about interaction process. This thesis contributes a visual language. The language proposed, the Kinetic InterActing language, (KIA) is composed of visual primitives, rules for composing expressions, and rules for interpreting these expressions. The KIA language is one possible instance of a visual language based on human movement constructs. - Design guidelines for visualisation of information about the interaction process. These guidelines are developed and followed in the design of visual language to reinforce and provide support about interactions based on KIA visual primitives and their behaviour. The guidelines support flexible, extendable, and modular features for visual languages. - A system of analysis. The system relates behaviour of visual primitives to visual patterns and the rules of their interpretation. The levels of analysis comprised from a number of different modes include: element and production analysis, range element and production range element analysis, and quality analysis. - Demonstration of the approach on case studies in healthcare. The demonstration of the language is presented over a number of case studies in the area of healthcare. To provide a demonstration of the interaction language the behaviour of the language elements in the computational representations are compared to expert evaluations over the same interactions. An overview of the computational processes supporting KIA is provided. Summary of the Results of the Demonstration of the Approach This thesis advances a methodology based on the premise that physical systems can be modelled as conceptual spaces. That is, modelling one system in action, interactions, through another system in action, human movement, is beneficial for improving the communicative value of interactions. The approach tackled a problem that is largely subjective and developed a formalism that provided the means to capture measures of a complex phenomenon. The use of metaphor has been extended in interpreting interactions through various components. The conceptualization and operationalisation of a framework for representing interactions provides analysis capabilities that through various measures generate in-depth semantics. The approach has been demonstrated with the development and application of a new visual language, KIA to a number of case studies in the domain of health. The results demonstrate the analysis of each case study based on KIA semantics. The feasibility of the language is demonstrated through comparison across the case studies and by comparing evaluation of interactions across five case studies by an expert and by KIA. The results support the hypothesis that representations of the interaction process provide benefit as to the dynamics of the interaction communicated through the behaviour of the constructs, elasticities and qualities. Consistency in semantics is achieved through the conceptual modelling of one system in action through another that inherently provides synergy. Through synergy a single command can set in motion a network of combined actions that constitute coordinated action in the system. Thus the achievement of the formalism is to communicate parameters of interaction through perceivable flow characteristics modelled as embodied information processes. The visual patterns provide access to information that can be utilised to improve the quality of the interaction. This thesis demonstrates that the major outcomes of the approach support the hypothesis; that to represent interactions so interacting parties can formulate meaningful interpretations of the unfolding of the interaction process is achievable through embodied information processes. Thesis Outline The thesis starts with an introduction to the problem formulation and the phenomenon of interaction. Chapter 1 introduces the issues of representing interactions and discusses the rational for the approach taken in this thesis. Chapter 2 provides the background for the approach taken and provides a critique of related work in the literature. The main components from the problem domain, interaction models, and interactions in healthcare are discussed. In Chapter 3, the methodology for deriving key constructs from human movement over contact improvisation video data is formulated and explained. Key constructs are sourced from the behaviour of the system under observation. Chapter 4 focuses on the modelling of interactions providing a system framework that covers high-level links of the key concepts involved in modelling interactions. Chapter 5 contributes a visual language, KIA. The KIA language is one possible instance of a visual language based on human movement constructs. Additionally design guidelines for visualisation of information about interaction process are discussed. Chapter 6 demonstrates the visual language through its application to a number of case studies in the domain of healthcare. Reliable evaluation is achieved by method of comparing expert and language evaluation for the case studies. Chapter 7 presents some concluding remarks and directions for future work. Issues with the approach taken are discussed. We identity the most promising areas for future application with, e-health, interactive environments, and virtual worlds. Suggestions are made as to how the approach and language developed in this thesis could be applied and extended. Key ideas are summarised at the end of each chapter and contributions both fundamental and practical are outlined.	en_US
dc.format	Thesis (PhD)	en_US
dc.language.iso	en	en_US
dc.rights	The author owns the copyright in this thesis including all reproduction and reuse rights for the work. The work may not be altered without the permission of the copyright owner. Attribution is essential when quoting or paraphrasing from this thesis.
dc.rights	info:eu-repo/semantics/closedAccess
dc.title	Human movement as a framework for developing computational representations of interactions	en_US
dc.type	Thesis
utslib.copyright.status	closed_access

Abstract:

NO FULL TEXT AVAILABLE. Access is restricted indefinitely. ----- Overview This thesis establishes a methodology for designing formal representations, visual languages, and visualisations of information about interactions. The conceptual modelling that enables the method for designing representations of interactions follows the elements of human movement. The representations of interaction process are modelled on bodily knowledge with the explicit objective of enhancing the semantics of the unfolding of interaction process. Interactions play a central role in our life. We interact with our colleagues and professional team members in the office, with our doctors during the visits to healthcare practices, with our lecturers and classmates in university. Through the interactions with the various computing systems that constitute our contemporary environment, we learn about it and achieve our goals. Interactions are situated at the very heart of the activities performed in many computerised domains, yet they remain poorly understood. Problem Formulation Each interaction constitutes a process that unfolds. This is the view the thesis takes. The way interactions unfold informs us about many aspects of the interactions themselves. Hence, in order to improve processes that rely on interactions, it is important to develop methods of encoding the way interactions unfold, capable of representing that information and allowing both humans and machines to utilise such information in order to improve interactions and, consequently, the processes that rely on them. For example, in health care, the overall process of patient treatment is a process that relies on interaction. It involves a series of interactions between patient and practitioner. If we consider such interaction in the context of the treatment process, then the overall treatment process will benefit. This research is about interaction, the interaction process, representation of such a process, and the manner in which bodily knowledge can be applied to the modelling of interactions. Interactions are complex, varied and dynamic. They are situated ‘transactions’ that derive meaning from the context within which they are embedded. The thesis addresses the problem of representing information about the interaction process that are machine usable. It seeks to develop a representation that allows capturing information about the unfolding of interactions. A central modelling issue is how to develop a representation of interactions that communicates meaning about the process without obscuring it. As it is undesirable to ‘obscure’ the representation of interaction the choice has been made to observe the act of interacting through the system of human movement. Human movement provides us with two important components for representing interactions; perception-action coupling and dynamics. This is suitable for the purposes because human movement by definition is a ‘reflective system.’ Through the output of this observation a representation of interaction can be defined. If the representation is modelled through the interacting processes that are integral to the representation of human movement interaction, then, by definition, the representation will partake of the same, or similar, characteristics as the system from which it was derived. There is then no a priori structure imposed on the construction of the interaction. Rather the phenomenon of interaction itself is the representation constructed. This methodology promotes a distancing of the work from the inherent subjectivity associated with content analysis approaches to representing interactions. Such models look to pre-defined selected features of interactions to provide definitions. In contrast, the approach in this thesis is to study the essential behaviour of interacting. Rather than starting with predefined notions of interaction, intuitive understandings of interaction based on experience form the initial basis. It is important to understand what interaction is, or what it is not, before any representation of interaction can be modelled. The general trend has been to consider interactions in terms of output, the results of the interaction, with little emphasis on the processes that construct interactions over time. There are problems with this approach as interactions are reduced to terms of output only and substantial part about them, the characteristics of the process of how they unfold, is not captured. Consequently the formalism developed in this thesis addresses this problem by capturing information about interactions that; - will improve the overall process, as it will allow us to extract knowledge about the way interactions unfold and incorporate that knowledge into the process; - will allow different views of that information to be accessed; - present, how such interactions unfold so that judgements can be made as to whether or not there has been a good communication. If provided with such profile of previous interactions in the beginning of a session, one can predict whether the current interaction unfolds well, and if not, based on the previous reflections, can respond with a strategy that can improve the interaction. The mentioned developments made in this thesis offer a range of contributions as summarised below. Thesis Contributions This thesis establishes a methodology for designing formal representations, visual languages, and information about visualisations of the interactions process. The conceptual modelling that enables the method for designing representations of interactions is inspired by Lakoff and Johnson’s approach to metaphors (Lakoff and Johnson, 1980). It allows expressing the target domain through the constructs of the source domain. In this thesis the source and target domains are human movement and interaction processes respectively. In the process the source domain is interpreted and formalised through well founded models of the system of human movement developed in movement observation science (Newlove & Dalby, 2004). From this are derived the constructs of elasticities and qualities. The behaviour of these two groups of constructs provides the shaping affinities for building expressions of interactions in the target domain. The mapping takes concepts that describe two frames of reference of human movement: (i) body position (the place of the body in space); and (ii) body dynamics (the motion that causes and expresses change from one position of the body to another), and uses their computational representations in the interactions domain. The set of constructs considered in this thesis include the following elasticities - the rising and sinking (RS-) elasticity and the contraction and extension (CE-) elasticity. The behaviour of elasticities describes the reciprocal effects between interacting parties and compactly expresses patterns of interaction. Below is a brief description of the thesis contributions. - Conceptual modelling of operating through the framework of metaphor between two physical systems, in particular human movement and interactions. The conceptual modelling establishes links between two physical systems by modelling one of them through the integral dimensions of the other. In the thesis, interactions as a system are modelled through the integral dimensions of human movement. - A methodology for modelling interactions. The methodology provides an analysis and transfer of the integral dimensions from one domain to another. Starting with observation of human movement, specifically over improvisational dance, it is then possible to derive concepts to model a formalism that behaves in a same manner. The methodology, called the InteractionSystem methodology, provides key insights of bodily knowledge; the way humans structure behaviour through bodily reasoning, association, and memory. From the observation of the phenomenon of interaction through human movement, the key constructs - elasticities and qualities - are derived. - A framework for interpreting interactions as a physical system. An interaction can be understood as a system through its components and the relations between them. This thesis has developed a multi-layer framework that provides key concepts of the system. - A language for expressing information about interaction process. This thesis contributes a visual language. The language proposed, the Kinetic InterActing language, (KIA) is composed of visual primitives, rules for composing expressions, and rules for interpreting these expressions. The KIA language is one possible instance of a visual language based on human movement constructs. - Design guidelines for visualisation of information about the interaction process. These guidelines are developed and followed in the design of visual language to reinforce and provide support about interactions based on KIA visual primitives and their behaviour. The guidelines support flexible, extendable, and modular features for visual languages. - A system of analysis. The system relates behaviour of visual primitives to visual patterns and the rules of their interpretation. The levels of analysis comprised from a number of different modes include: element and production analysis, range element and production range element analysis, and quality analysis. - Demonstration of the approach on case studies in healthcare. The demonstration of the language is presented over a number of case studies in the area of healthcare. To provide a demonstration of the interaction language the behaviour of the language elements in the computational representations are compared to expert evaluations over the same interactions. An overview of the computational processes supporting KIA is provided. Summary of the Results of the Demonstration of the Approach This thesis advances a methodology based on the premise that physical systems can be modelled as conceptual spaces. That is, modelling one system in action, interactions, through another system in action, human movement, is beneficial for improving the communicative value of interactions. The approach tackled a problem that is largely subjective and developed a formalism that provided the means to capture measures of a complex phenomenon. The use of metaphor has been extended in interpreting interactions through various components. The conceptualization and operationalisation of a framework for representing interactions provides analysis capabilities that through various measures generate in-depth semantics. The approach has been demonstrated with the development and application of a new visual language, KIA to a number of case studies in the domain of health. The results demonstrate the analysis of each case study based on KIA semantics. The feasibility of the language is demonstrated through comparison across the case studies and by comparing evaluation of interactions across five case studies by an expert and by KIA. The results support the hypothesis that representations of the interaction process provide benefit as to the dynamics of the interaction communicated through the behaviour of the constructs, elasticities and qualities. Consistency in semantics is achieved through the conceptual modelling of one system in action through another that inherently provides synergy. Through synergy a single command can set in motion a network of combined actions that constitute coordinated action in the system. Thus the achievement of the formalism is to communicate parameters of interaction through perceivable flow characteristics modelled as embodied information processes. The visual patterns provide access to information that can be utilised to improve the quality of the interaction. This thesis demonstrates that the major outcomes of the approach support the hypothesis; that to represent interactions so interacting parties can formulate meaningful interpretations of the unfolding of the interaction process is achievable through embodied information processes. Thesis Outline The thesis starts with an introduction to the problem formulation and the phenomenon of interaction. Chapter 1 introduces the issues of representing interactions and discusses the rational for the approach taken in this thesis. Chapter 2 provides the background for the approach taken and provides a critique of related work in the literature. The main components from the problem domain, interaction models, and interactions in healthcare are discussed. In Chapter 3, the methodology for deriving key constructs from human movement over contact improvisation video data is formulated and explained. Key constructs are sourced from the behaviour of the system under observation. Chapter 4 focuses on the modelling of interactions providing a system framework that covers high-level links of the key concepts involved in modelling interactions. Chapter 5 contributes a visual language, KIA. The KIA language is one possible instance of a visual language based on human movement constructs. Additionally design guidelines for visualisation of information about interaction process are discussed. Chapter 6 demonstrates the visual language through its application to a number of case studies in the domain of healthcare. Reliable evaluation is achieved by method of comparing expert and language evaluation for the case studies. Chapter 7 presents some concluding remarks and directions for future work. Issues with the approach taken are discussed. We identity the most promising areas for future application with, e-health, interactive environments, and virtual worlds. Suggestions are made as to how the approach and language developed in this thesis could be applied and extended. Key ideas are summarised at the end of each chapter and contributions both fundamental and practical are outlined.

Please use this identifier to cite or link to this item:

http://hdl.handle.net/10453/35970