Involuntary movements, such as tremor, tics, sudden jerks (chorea) and muscle spasms (dystonia) occur in many neurological disorders. These involuntary movements are difficult to treat, requiring medications (which often have unpleasant side effects) or invasive deep brain stimulation (requiring electrodes to be implanted in the brain). Recent innovative research by Prof Stephen Jackson (UoN Psychology) found that non-invasive median nerve stimulation (MNS), which involves stimulating a nerve in the wrist using a wearable electronic device, suppresses involuntary movements in Tourette syndrome by entraining oscillations in relevant brain circuits when the device is active. Clinical trials are now underway, or being developed, to test the therapeutic effect in a number of neurological disorders, including Parkinson disease (PD) ataxia telangiectasia (A-T) and restless legs syndrome (RLS). 

However, involuntary movements are intermittent and highly variable, meaning that continuous stimulation by the MNS device is not required or desirable. Real-time detection of involuntary movements could allow personally-tailored therapeutic stimulation strategies. optimised to detect different types of involuntary movements seen in diseases such as PD, AT and RLS, and used to optimise the stimulation regime on an individual basis. 

In this project, we will analyse videos of trial participants to understand how MNS affects the patients' symptoms, and to optimise the device in real time to provide the maximum benefit for each individual. To achieve this, we will utilise state-of-the-art marker-less pose estimation in combination with multi-modal machine learning to better understand the complex movements of those with movement disorders. This will enable us to tailor the treatment to each individual's specific needs in real time and maximise its effectiveness by tuning the MNS device. Demonstrating the feasibility of the human-in-the-loop approach will directly enable clinical trials of the effectiveness of personalised home-administered MNS stimulation in reducing the unwanted movements, and allow exploration of the potential benefits of this technology in improving quality of life for individuals with movement disorders. 

This PhD project will benefit from a strong multidisciplinary approach, combining computer science, psychology, and neuroscience. Applicants are expected to have a combination of programming experience (python) and an interest/background in neuroscience. 

Supervisors: Dr Alexander Turner (School of Computer Science), Prof Stephen Jackson (School of Psychology), Prof Robert Dineen (Faculty of Medicine & Health Sciences) 

For further details and to arrange an interview please contact Dr Alexander Turner (School of Computer Science)

The first five years of the child’s life play a critical role for cementing cognitive functions. Several studies have shown that the quantity and quality of parent-child interactions can affect children’s cognitive development later on. Periods of shared attention between caregivers and children have important implications for developing the child’s attention span and language skills. Recent neuroimaging research has also found that some parts of the brain may be activated in similar ways for both parents and children during these interactions, and interestingly, the extent of this analogous brain activation might be influenced by factors like how stressful the home environment is. Despite what is known about the association between child cognitive functions and the modalities (i.e., audio-visual and brain activity) involved in parent-child interactions, the extent to which these can be combined to inform better cognitive developmental outcomes for infants is unknown. If we can better understand these associations, we can help caregivers interact with young children in more effective ways that could potentially transform their development in the crucial early stages of life. 

In this project you will work to better understand the interactions between young children and caregivers as they interact during exploratory play. You will apply state-of-the art machine learning techniques to analyse videos of interactions, detecting poses, activities and key events. You will explore novel deep learning methods for integrating multi-modal information sources, to combine video events, audio data and FNIRS data and extract perceptual, verbalization, affect, brain function information (to name a few) during parent-child interactions to predict cognitive functions in children. Video, questionnaire and neuroimaging data are already available from an ongoing, longitudinal project assessing neurocognition in children in the School of Psychology at the University of Nottingham. Separately, it might be possible to design and collect more data in the future. 

Applicants will be expected to have a good working experience with current machine learning and image processing tools and techniques. Prior knowledge of biomedical signal processing and natural language processing is desirable but not essential. 

Supervisors: Dr Joy Egede (School of Computer Science), Dr Sobana Wijeakumar, (School of Psychology), Dr Aly Magassouba (School of Computer Science) aly.magassouba@nottingham.ac.uk. 

For further details and to arrange an interview please contact Dr Joy Egede (School of Computer Science)

This research will address the design and evaluation of safe and trustworthy collaborative robots in assistive scenarios. 

Robotics and autonomous systems (RAS) are emerging as disruptive technologies with the potential to provide personalised and cost effective support for a range of care-related tasks for people with disabilities, including the provision of physical and social assistance and physiotherapy. However, in order to ensure real-world deployment and commercialisation, application-focussed research into safe Human-Robot Interaction, Hazard Analysis and Risk Assessment in a range of dynamic environments and scenarios of use is needed. 

This imperative for these areas of research is particularly significant, given the vulnerability of the end-users interacting with these systems – giving rise to a range of very complex safety and reliability issues and concerns. It is necessary to carefully and deeply consider the safety of assistive robots at not just an operational and functional level – but also from human factors and clinical efficacy perspectives. 

The research will start with a user-centred approach to identifying safety-related issues of specific assistive robots to scope the requirements for real-world use assistive robots by people with different accessibility needs and contexts. As part of this you will also need to review the existing methods and approaches for safety assurance for these systems, with a view to exploring critical barriers to assurance and regulation. Through user-based testing and evaluation using existing assistive robotic platforms, you will analyse the adequacy of current guidelines and standards for assistive robots and identify gaps in the standards using a set of real-world use cases.

Based on your skills and interest, there are several routes you can also consider for this PhD, from the development and validation of hybrid implicit and explicit human-AI mechanisms for generating safe behaviours and conducting hazard analysis and risk assessment, to considering human-factors and psychology related issues that can impact safe interaction, or even experimenting with new forms of embodiment-based social signalling.

Prospective PhD applicants are expected to have a degree in Engineering, Computer Science or Maths with knowledge of Data Science, Machine Learning and AI. Applicants with a background in human-factors and psychology are also welcome. This project will require excellent programming skills with evidence of proficient working knowledge in one or more of the following: C++, C, Java, Python, ROS. 

Supervisors: Profs Praminda Caleb-Solly (School of Computer Science), Carl Macrae (Professor of Organisational Behaviour and Psychology)

For further details and to arrange an interview please contact Prof Praminda Caleb-Solly.

Physical activities such as walking, exercise and dance are not only good for physical well-being, but also mental well-being, particularly when done together with someone. 

This research will explore how robotics technology-based mobility devices could be designed and used to connect people in different remote locations to mediate and coordinate physical interaction between them for physical activities such as exercise, walking and dance.

There are number of avenues to consider in this research, such as exploring most intuitive, effective and engaging ways for people to coordinate their movement-based activities via their assistive robots when they are not in the same physical space, investigating what kind of personalisation methods and input/output modalities are useful to improve the interaction between humans through the robotics technology-based mobility devices and enable long term adaptation to changing needs, or in what ways are interactions affected by people's accessibility needs, their cultures, communities and the interaction environments, or what are suitable embodiments or form-factors for such devices.

This PhD project will benefit from a strong multidisciplinary approach at the interface of Computer Science, Robotics, and Physiotherapy and Dance. Applicants are expected to develop technological advancements in AI and Interaction Design, including using machine-learning for generating personalised user models for children and adults, adaptive motion planning in social environments, feedback generation. In addition, the successful student will design, conduct and analyse experiments to investigate the socio-psychological effects of the technologies. 

Supervisors: Praminda Caleb-Solly and Paul Tennent

For further details and to arrange an interview please contact Prof. Praminda Caleb-Solly. 

Partner: Screen South https://screensouth.org/ 

Telepresence robots offer a significant digital opportunity for people to remotely access social, work and cultural spaces, autonomously moving around them, giving a feeling of connection and presence. As such, telepresence robots can be a transformative tool in enabling engagement with museums and galleries, making connections and improving wellbeing. For a number of disabled people, and those shielding due to lowered immunity due to long-term conditions, having the choice to access cultural spaces and interact with people and objects through telepresence robots, can offer more freedom and flexibility to be ‘present’ in locations.  

However, the interfaces to control telepresence robots can be cumbersome and inaccessible, particularly for those with sensory and/or physical impairments, making it difficult or impossible for them to use these effectively. We are also interested in exploring how by combining telepresence robots with other digital devises, such as VR and haptics, we can enable truly immersive multisensory experiences that are accessible to a variety of participants. 

The aim of this research is to co-design and test a range of different input and output devices and modalities to develop multisensory interfaces that will enable accessible, smooth and enjoyable control and remote interaction. You will explore the integration and use of speech, head and ear-switches, electromyograms, and gaze, amongst other modalities, for control, and visual, haptic and aural modalities for feedback of information to enable rich and creative experiences of the remote space, people and objects. You will study and develop metrics for evaluating usability and user experience for accessible teleoperation using these modalities and custom devices, as well as developing a best practice framework to support future accessible design. The research will also offer the opportunity to draw on disability studies research to understand the lived experience of using telepresence in different contexts, understanding impact on self-efficacy, identity, social relationships and agency in interactions. This research offers several technical and non-technical strands to explore, based on the candidate’s background, skills and experience.

For further details and to arrange an interview please contact Prof. Praminda Caleb-Solly. 

Partner: Queen’s Medical Centre, University Hospital, Nottingham

In busy clinical environments, particularly where patients have a high-level of staff dependency, providing support for clinical staff to improve patient monitoring, triage and management can not only help to ease level of staff stress, but also potentially improve patient safety. This research will investigate how information to assist with clinical decision making can be presented through creative ambient and/or augmented information displays and the impact that different modes and modalities have on user cognitive load, attention and efficiency. This research is situated in the use of tangible devices, and ambient and augmented reality displays, exploring topics in information visualisation, sensory substitution, human factors and user experience design. Considering the context of high-pressured environments, such as dementia wards, you will begin the research with a qualitative observational study, scoping the requirements using co-design with clinical and care professionals, before designing, developing and evaluating a range of approaches for representing the required information.

Based on the candidate’s academic background, skills and experience, the research focus can be either on developing intelligent sensing to capture and represent the key information required for decision-making, or design and development of the approaches for displaying it through different means and modalities, or a combination of both.

For further details and to arrange an interview please contact Prof. Praminda Caleb-Solly. 

Partner: Robotics For Good CIC https://www.roboticsforgood.co.uk/ 

Assistive technologies, such as smart home environments, integrated sensors and service robotics are recognised as emerging tools in helping people with long-term conditions improve their quality of life and live independently for longer. A key aspect of the research into assistive robotics for assisted living is developing contextual and social intelligence for the robot to interact appropriately, safely, and reliably in real-time. This research relates to developing assistive robot behaviour by incorporating both environmental and user data, and behaviour, as part of an overall intelligent control system architecture.

In addition to having a ‘memory’ of previous interactions and situations, assistive robots need access to information that is current and one that provides a dynamic world view of the user (including their emotional state) so that they can provide information and responses that are contextually appropriate. Typical activities for which support can be provided is support with rehabilitation, medication management, cognitive and social stimulation, nutrition management etc. Drawing on information from environmental and activity sensors instrumented into a smart home, and information about the user’s current physical and emotional state, assistive robots can potentially create value through provision of interventions that are more socially intelligent regarding how, and what advice and support they provide. To create a more holistic service, that takes into consideration prioritisation of events based on aspects of health and social circumstance requires an adaptable, intelligent learning system. Building on existing research on intelligent control system architectures, the aim of this research will be to design and test modular semantic memory architectures that can be adapted over time. You will investigate optimal combinations of contextual data comprising implicit (emotional, physiological) and explicit user data (interaction), as well as behavioural activity data assimilated from a range of wearable and smart home sensors, to develop adaptive, intelligent and emotionally engaging robot behaviour to support independent living. 

Fully funded PhD Studentships in Human-Robot Interaction 

• • We have two fully funded PhD studentships for talented candidates to join us from the 1st of October 2024.  

  • You will recieve an annual tax-free stipend based on the UKRI rate plus fully-funded PhD tuition fees for the four years (Home/UK students only) 

  • The students will work with the Boston Dynamics Spot robot to solve real life inspection problems in extreme and factory scenarios. We will develop incremental learning methodologies to develop context-based policies, not only for navigation, but error recovery in long term automation. Human-in-the-loop and teleoperated control methods will be used as the backbone strategy to ensure increasing levels of autonomy during inspection. We will look at human-robot interaction methodologies for the day-to-day operation of the Boston Dynamics Spot mobile inspection robot.

  • The two projects will be in collaboration with RACE (https://race.ukaea.uk/) and Reckitt (https://www.reckitt.com/)

This PhD project will develop on the Nottingham Robotic Mobility Assistant, NoRMA (https://github.com/HCRLabRepo/NoRMA) to study triadic learning methodologies for developing effective assistance policies for wheelchair users to support their day to day activities.

This PhD project will be in collaboration with RACE (https://race.ukaea.uk/) and aims to develop incremental learning methodologies to develop context-based policies, not only for navigation, but error recovery in long term automation. Human-in-the-loop and teleoperated control methods will be used as the backbone strategy to ensure increasing levels of autonomy during inspection. We will look at human-robot interaction methodologies for efficient management and optimisation of parallel tasks encountered in day-to-day operation of the Boston Dynamics Spot mobile inspection robot.

This PhD studentship will investigate trust from a theory of mind point of view to model a robot’s trustworthiness from the perspective of a human, and vice versa.

Lifelong learning with robotic vacuum cleaners in social spaces: In collaboration with Beko Plc. (https://www.bekoplc.com/), this PhD project will focus on these challenges by targeting multiple strands of research in perception, planning, human-in-the-loop learning, and shared control for service robots. The ability to detect and recover from errors during navigation is an essential ability for an autonomous service robot that can run for extended periods of time. In addition, functioning in human settings, these robots should be programmed to adhere to social cues in a context- dependent manner, not only to enable safe, but also acceptable functionality.

Human-robot interaction (HRI) is a multidisciplinary field that studies how humans and robots can communicate and collaborate effectively and naturally. Gesture recognition is one of the key components of HRI, as it enables humans to use intuitive and expressive body motions to convey commands, intentions, and emotions to robots. However, most of the existing gesture recognition methods rely on vision-based sensors, such as cameras, that have limitations in terms of occlusion, illumination, privacy, and computational cost.

The aim of this PhD project is to develop novel methods for inertial sensor-based gesture recognition for HRI.

The project will involve the following objectives:

• Review the state-of-the-art methods and challenges of inertial sensor-based gesture recognition for HRI.

• Develop new methods for gesture segmentation, classification, and generation using inertial sensors, such as accelerometers and gyroscopes, worn on the human body.

• Evaluate the performance and usability of the proposed methods on various HRI scenarios and tasks, such as navigation, manipulation, social interaction, and entertainment.

• Investigate the human factors and ethical issues of using inertial sensors for gesture recognition for HRI.

The successful candidate will have a strong background in computer science, engineering, or mathematics, with good programming skills in Python or C++, and good knowledge of machine learning. Experience in inertial sensor data processing, machine learning, or human-robot interaction is desirable but not essential. The candidate will be supervised by Dr. Lucas Fonseca and Prof Praminda Caleb-Solly  from the School of Computer Science, and will have access to the state-of-the-art facilities and resources of the CHART research group.

Human-machine interfaces (HMIs) are systems that enable humans to interact with machines, such as computers, robots, or assistive devices, using various modalities, such as speech, touch, or gesture. HMIs have many applications, such as entertainment, education, health, and industry. However, most of the existing HMIs are based on small and predefined set of actions, which limit the naturalness and expressiveness of the human-machine interaction. In addition, they often don't consider the user's limitations.

The aim of this PhD project is to explore the use of human movement as a strategy for designing and evaluating novel HMIs that can adapt to the user’s preferences, context, and goals. 

The project will involve the following objectives:

• Review the state-of-the-art methods and challenges of using human movement for HMIs.

• Develop new methods for capturing, analyzing, and synthesizing human movement data using various sensors, such as inertial sensors, motion capture systems, or cameras.

• Design and implement novel HMIs that use human movement as an input or output modality for various tasks and domains, such as gaming, education, or rehabilitation.

• Evaluate the usability and user experience of the proposed HMIs using quantitative and qualitative methods.

The successful candidate will have a strong background in computer science, engineering, or design, with good programming skills in Python or C++. Experience in human movement analysis, machine learning, or human-computer interaction is desirable but not essential. The candidate will be supervised by Dr. Lucas Fonseca and Prof Praminda Caleb-Solly  from the School of Computer Science, and will have access to the state-of-the-art facilities and resources of the CHART research group.

Context: Stroke is a major cause of disability, with nearly two-thirds of survivors facing some form of disability post-hospitalisation. The disabilities resulting from stroke encompass physical and cognitive impairments, necessitating long-term care support. A common and often devastating condition resulting from stroke is aphasia, a language impairment that impedes a person’s communication and social interaction. Healthcare solutions must also support social reintegration of patients with aphasia for their mental well-being.

Project: This project consists in developing a multimodal communication models for aphasia patients, leading to a new generation of intelligent speech aids. Classic applications using visual aids and pictures methods often lack robustness and are not flexible to adapt to real-world environments or evolving user needs. Using methods of multimodal language understanding, the goal is to create a multimodal AI model to comprehend impaired language in daily living activities. 

Given the strong multidisciplinary context of this project, applicants are expected to have skills in natural language processing and AI.

Context: An emerging research area is the robot-assisted rehabilitation after surgery or stroke. After surgery or after a stroke, it is often beneficial from a medical perspective to move the patient’s joint and muscles intensively over a sustained period to improve healing. If this mobilization does not take place, complications may arise, such as restricted mobility at a later stage. This requires moving the patient’s joint and muscles to accelerate patient recovery.  Such a task can be represented as the manipulation of a multi-body system (e.g. articulated objects). 

Project: To this end, this project will rely on collaboration work with the University of Bordeaux, France which has led to a groundbreaking AI approach for manipulating deformable linear objects (DLO) [1]. In this work, a new multi-agent deep reinforcement learning framework was developed to manipulate DLO of different characteristics into a desired shape. 

Based on this algorithm, the proposed project will consist in extending the above work to articulated limb manipulation. This will require: 1) integrating tools like OpenSim, simulating biomechanical systems into the robot training environment (PyBullet or IsaacSim) 2) developing new policies and reward functions related to the limb configuration. 

Prospective PhD applicants are expected to have a degree in Computer Science with knowledge in data science and machine learning.  Applicants should also be proficient in computer programming (Python, C++, etc.).

[1] M. Daniel et al., "Multi Actor-Critic DDPG for Robot Action Space Decomposition: A Framework to Control Large 3D Deformation of Soft Linear Objects," in IEEE Robotics and Automation Letters, vol. 9, no. 2, pp. 1318-1325, Feb. 2024  

This project will be supervised in collaboration with Pr. Lisa Yon from the School of Veterinary Medicine and Science Experience, within the Elephant Welfare Project.

Context: Elephants are the largest land-dwelling animal; Asian elephants in the wild walk up to 21 km/day, and African elephants walk up to 28 km/day.  This high level of mobility is essential for good joint health. But most zoo elephants live in restricted spaces, and cold climates that require extended periods of indoor confinement, so they cannot be as active.  Inactivity and high food intake predisposes captive elephants to develop osteoarthritis (OA).  OA is an inflammatory condition of the joints resulting in wear and tear and bony abnormalities, and is a major cause of chronic lameness in captive elephants. Early detection of OA is essential, in order to implement effective treatment to address inflammation and pain, but also to attempt to halt disease progression.  However, due to the sheer size of elephant limbs, early methods of detection such as radiographs are not always feasible, and detection of lameness and pain in elephants through visual analysis of gait is currently quite unsuccessful, such that joint inflammation and pain is often only detected quite late in the progression of the disease.

Project: This research will develop multimodal models for musculoskeletal diseases detection (MSD) by combining perception sensing (e.g., from video footage, rumble recording or wearable device) with behavioural assessments in the form of structured data or language input. This research will identify the behavioural observations, gathered from elephant carers, that can be linked to MSD. The technical challenges lie in developing models that can scale and remain reliable, both temporally and spatially, when dealing with diverse and sequential real-world data.  These challenges will be addressed by drawing inspiration from foundational models, in particular multimodal large language models (MLLMs), to efficiently combine different modalities.  This will allow for individualised AI models, enhancing their effectiveness and applicability.

Given the strong multidisciplinary context of this project, applicants are expected to have or develop skills in AI, perception, and language understanding. Prospective PhD applicants are expected to have a degree in Computer Science with knowledge in data science and machine learning.  Applicants should also be proficient in computer programming (Python, C++, etc.). Experience in wildlife science is not mandatory but appreciated. 

Context: During an interaction, robots should not only understand the user's intent, but they should also be able to communicate in a comprehensive way about their actions and decisions. Bidirectional communication favors interaction and trust towards robots. Paradoxically, despite the recent success of robot learning methods, current neural network models are not adapted to this paradigm as they are inherently black boxes. For this reason, the research topic related to explainable AI (XAI) has recently been established. The purpose of XAI is to develop white-box neural networks that can be interpreted and understood. This approach has been particularly used in the computer vision community, with methods such as class activation maps or the attention branch network providing an explanation through visual attention. However, all these approaches have been limited to the perception level so far and have not been grounded in the physical world with robots.

Project: In this Ph.D. project, the applicant will develop models for linguistic explanation by considering multimodal input related to the state of the operator, robot, and environment and the task being performed. These models would generate a set of sentences that describe the actions / decisions taken by the robot, such as “I cannot reach the blue bin. You are on my path, and I might collide with you”. To generate such a sentence, network architectures based on transformers and state-of-the-art LLM, combining supervised and reinforcement learning will be developed. Additional applications will be derived from the above approach. More specifically, a summarizer engine will be developed to report all the tasks performed by a robot. An ergonomics/safety recommendation engine will also be developed to warn the user about possible hazardous motion/behavior considering human activity.

Given the strong multidisciplinary context of this project, applicants are expected to have or develop skills in AI, perception, and language understanding.

Context: The adaptable, human-assistant paradigm in telerobotics and telepresence applications becomes even more useful as well as complex when we consider interfaces for expert as well as non-expert users, especially in multi-person/multi-robot remote telepresence scenarios. Robotics provides an advanced solution to mitigate risks in extreme work environments (e.g., nuclear, disaster response, etc.), through technologies such as remote telerobotics, advanced haptics master devices, and smart sensing and visualization. This project will develop new software and hardware systems for an immersive 3D user interaction experience for interfacing with robotic systems (e.g., Franka Emika Panda robot arm, Boston Dynamics Spot robot, etc.) The project will use, develop, and integrate advanced technologies in VR / AR / MR towards improving the situational awareness of the operator(s), providing an intuitive and intelligent user interface for robotic teleoperation and monitoring in high-risk environments. One of the ways of achieving that is using natural language interaction (NLI) to increase the robot autonomy and user assistance in teleoperation tasks, thereby reducing the cognitive burden on the user. For e.g., “grasp the cup in the bottom half for better grip” is a natural language command that the robot should be able to execute autonomously. Such NLI can also help improve deep learning-based semantic scene understanding and object tracking outcomes and accuracies. Visualized scenes can be represented with generated mesh models in the MR interface using high-level encoding, i.e., real-time text-to-scene description and 3D processing (“the vessel at the top right of the field-of-view feels softer than the one on the left”, “the object at the bottom is a chair with a broken arm lying on its side”, etc.). 

The project will build on the strong existing technological capabilities in the CHART group, acquired through the successful implementation of high-tech projects in this field. During this program, the student will develop and utilize their knowledge in: 

• Real-time 3D reconstruction and tracking of dynamic remote scenes and objects; 

• Real-time rendering of complex remote information in an immersive MR interface;

• Deep learning for semantic remote scene understanding

• Natural Language Interaction using LLMs

Project: This PhD project will benefit from a strong multidisciplinary approach at the interface of Computer Science and Robotics and will focus on evaluating the usability and user experience aspects for accessible telerobotics using mixed reality. Prospective students should have a degree in Computer Science, Engineering, or other related fields, and would be beneficial to have relevant competencies in computer vision, coding (C/C++, Python), deep learning algorithms (YOLO, TensorFlow), VR software (Unity, Unreal Engine), and VR hardware devices (HTC Vive, Meta Quest), etc. Experience with robotic software (ROS2, Gazebo) and hardware (manipulators, mobile robots) is a plus!