This paper suggests a biomechanical analysis of the joint motion of the arm based on the 3-dimensional motion analysis technique in the steering wheel action. The three-dimensional motion trajectory of the arm is obtained using the motion capture system, and the inverse kinematic analysis and inverse dynamics analysis are performed on the musculoskeletal model of the arm using this result. By doing this, the displacement of the arm joint and the magnitude of the joint torque are calculated. The rotational motion of each joint occurs simultaneously and in the joint motion, it can be confirmed that pronation-supination of the elbow joint, adduction-abduction of the shoulder joint, flexion-extension and flexion-extension of the wrist joint are the main movement parameters. 

Using this biomechanical motion analysis of joint segmental range of motion, it can be extended to analyze the behavior of passengers indoors in the autonomous vehicles.

In the autonomous vehicles, it is necessary for the driver to analyze the driver's intentional/unintentional behavior and perception according to changes in traffic conditions. Most of the steering systems currently used have a function of assisting the power, so hydraulic or electric driving force is used to enable the driver to smoothly operate the steering wheel with little force. It provides convenience features. As vehicles become more intelligent and electronic, the recently developed steering system has evolved to a level that not only provides a function of assisting steering power, but also performs active functions such as controlling steering torque or changing lanes by judging by itself according to driving conditions. is becoming These include drive-by-wire systems, active steering systems, and lane keeping systems. However, as a future car, autonomous cars focus on providing convenience to occupants, so the driver can use the steering system depending on the driving situation. should be able to provide Therefore, it is necessary to examine the behavior of the driver's upper body in order to understand the driver's intention for the driving situation. Accordingly, it is necessary to quantify and analyze the range of motion for each joint segment of the driver's upper body.

The steering system transmits steering input to the wheels and at the same time feeds back the tire traction that changes depending on the driving situation to the driver. It is a man-machine system. Until now, research on the steering system has mainly focused on hardware or logic to control it, and the driver's behavior has not received much attention. However, in order for the intelligent steering system to have optimal performance, an approach that connects the driver to the hardware of the steering system is required. From this perspective, this study aims to understand the characteristics of the driver's movement by performing a biomechanical analysis of the driver's arm joint motion in the steering motion as the simple example.

In order to interpret the movement of the musculoskeletal system of driver, it is essential to understand the muscle movement that causes the joint movement between bones. Delp [1] defined muscle-tendon parameter values ​​through analysis of young cadavers, and developed a human motion analysis program based on this. Garner and Pandy et al. [2,3] studied 26 major upper body muscles and movements through an optimization technique based on maximum muscle force, muscle length, and tendon length in an isometric state. Manal and Buchanan [4] developed a numerical optimization technique for tendon length under the assumption that the change in muscle length is greater than the change in tendon length. A biomechanical analysis of steering motion was performed using a 7 DOF arm motion model [5~8]. Through the musculoskeletal model of the arm connected to the steering device, the joint torque applied to the shoulder, elbow, and wrist during steering motion was theoretically calculated. Recently, motion analysis techniques, which are rapidly developing in terms of measurement equipment, signal processing technology, and biomechanical modeling, enable reliable interpretation of the musculoskeletal system movement of the human body.

By attaching a marker that reflects light to the human body, we are recording continuous motion using a motion capture camera, obtaining the three-dimensional trajectory of each part of the human body through image processing, and inputting the result into the musculoskeletal model of the human body joint. It allows to calculate the kinematic behavior of the muscle and the muscle force required for it. This is to analyzes the range of motion (ROM) and motion characteristics of the driver's shoulder, elbow and wrist joints during steering operation using motion analysis technique. Using this interconnected model between the musculoskeletal model and motion capture model of human body, the torque transmitted on each joint during steering operation is estimated. In addition, this study can be to present a basis for analyzing the sensitivity of joint motion according to the change of the relative position and angle of the driver's motion and the steering system with respect to the driver's seat. This paper is to introduce a general and physically intuitive system of the indoor occupant representation of autonomous vehicle which may be used qualitatively and quantitatively. Contrary to the simplistic situation in Fig. 1 given the complexities of the human actions indoor of autonomous vehicle, we would need several quantities for its complete description. In a multi-dimensional space, such quantities can be represented as a point which characterizes the indoor actions. Different indoor actions are represented by different points and the evolution of an action can be characterized by a locus of points in that space. From a dynamic systems point of view, a complete set of “state variables” uniquely describe a system. For example, the joint angle and the joint velocity constitute the state variables of multi-bodies and can completely represent its motion. Although the level of our current knowledge does not permit us to search for the complete set of state variables of the complex dynamics called the human actions, we nevertheless extract some measurable, communicable and meaningful action descriptors to describe the occupant actions. These occupant action descriptors available in the indoor of autonomous vehicle may reflect the changes in action pattern in response to some externally controllable factor or parameter. 

The schematic configuration of the experimental device for motion analysis is shown in Fig. 1, and the driver's seat simulator for a passenger car used in the experiment is manufactured as shown in Fig. 2. The driver's seat can change the front and rear position with respect to the reference point and the angle of inclination of the backrest, and the angle and height of the steering wheel can be adjusted. In order to simulate the steering reaction force of the steering wheel, the servomotor connected to the steering shaft generates the steering reaction force as a function of the steering angle. The magnitude of steering reaction force assumes the torque required to steer the hydraulic power handle for a general passenger car. Steering torque and rotation angle are measured by a torque meter and rotational displacement sensor. For motion analysis, a marker is attached to the driver's body, and the movement of the marker is tracked using four motion capture cameras. The trajectory of marker is converted into three-dimensional data using real-time signal processing software. The rotation angle and joint torque of the shoulder, elbow, and wrist joints are calculated using the musculoskeletal system modeling software of the human body. 

For steering motion analysis, there should be an integrated model in which the driver's musculoskeletal model and the steering system's mechanical system are linked. Since the steering motion is mainly generated by the movement of arm joint, it is assumed that the upper body is fixed to the seat and only the arm part from the shoulder to the hand is considered. As the motion of arm model is shown in Fig. 3, it consists of the upper arm (between the shoulder and the elbow), the forearm (between the elbow and the wrist) and the hand. The movement of the upper arm is three degrees of freedom: Flexion-Extension, Abduction-Adduction, External Rotation-Internal Rotation at the shoulder joint, and the movement of forearm is at the elbow joint. Two degrees of freedom for flexion-extension, pronation-supination, and the hand has 2 degrees of freedom for flexion-extension and radial deviation-ulnar deviation at the wrist joint. It was assumed to have a total 7 degrees of freedom. [5~8]

In order to obtain the 3-dimensional relation between the arm posture and the non-linear steering response in Fig. 4, the joint torques and angles should be known. Hence, a 3-dimensional model of the human posture needs to be developed to determine the joint angles for different steering angular positions, and finally to compute the joint torques. The following assumptions are considered for the human body model: 

• A quasi-static model is implemented to ensure a unique solution, because only the steady-state behavior is investigated.
• Shoulder and elbow joints are considered, neglecting wrist because of its negligible influence in terms of displacement and force on the overall system.
• In the shoulder joints only x- and y-rotation are considered. This ensures sufficient reachability while not including extra degrees of freedom and realizing a less computationally demanding model.
• The shoulder joint rotation in x-direction is fixed to a constant value to ensure a unique quasi-static solution. This is motivated by the assumption that these angles vary the least compared to the other joint angles during steering.
• The elbow joints are limited to moving in y- and z-direction only to ensure a unique solution while retaining the required reachability on the steering wheel trajectory.
• The mass between the shoulders is fixed in x-, y- and z-direction at the center of mass.
• The endpoints of the forearms (i.e., the hands) are fixed to the steering wheel trajectory in a quarter-to-three steering posture.
• The upper arm is constrained which prevents independent movement with respect to the mass between the shoulders.
• Forearm and upper arm are constrained so they are not allowed to move independent from each other.
• The centers of mass are positioned according to anthropometric standards [9] and, therefore not located exactly in the middle of each body. All other parameters describing limb lengths and masses follow the same anthropometric standards.

Figure 1: Schematics for driver’s steering motion analysis

Figure 2: Experimental steering simulator

Figure 3: Arm joint motion

Figure 4: Driver-steering system interactive model [9]

Motion analysis calculates the angle at the joint center using the position information of marker measured in the motion analysis experiment, and calculates the angular velocity and angular acceleration. Then, after constructing the equation of motion using multi-body dynamics analysis software, the joint moment is calculated from the angular acceleration information of the three joints, and the change in the attachment position of the muscle and the moment arm are calculated. The calculated joint moment at each time step can be composed of a passive joint moment component generated from ligaments and cartilage in the joint and an active joint moment component due to muscle strength generated by contraction of the arm muscles. The passive joint moment is greatly affected by the subject's anatomical shape, the stiffness and shape of the joint soft tissues (ligaments, cartilage, etc.), and the attachment position. The empirical formula was used as suggested in ref. [10]. A quasi-static optimization formula is implemented to distribute the active joint moment to the arm muscles.[11] In the optimization formula, the lower extremity muscle strength becomes a variable, the physiological maximum strength of each muscle is limited [12], the constraint conditions are corrected so that the moment at which the muscle strength occurs matches the active joint moment calculated in the dynamic analysis. The objective function is the sum of the three powers of the muscle stress divided the muscle load by the muscle cross-sectional area.[11]

In this experiment, the position where the driver grips the steering wheel is assumed to be the 2 o'clock and 10 o'clock positions of the steering wheel. The driver rotates 120 degrees counterclockwise from the initial position at a rotation speed of 60 deg/s, then returns to the initial position, and then rotates 120 degrees clockwise again. At this time, make sure your hand does not miss the handle. The steering reaction force of the steering wheel is given by measuring the steering reaction force generated when a typical passenger vehicle changes lanes at a speed of 80 km/h, and as shown in Fig. 5, the actually measured steering torques as a function of the steering angle are compared with the simulated ones. It can be seen that the magnitude of torque and the hysteresis characteristics are being simulated. Since the movements of the left and right arms are symmetrical to each other during steering, motion analysis is performed only on the right arm. The rotation angle of each joint is a value measured based on an angle of 0 degrees in an anatomical position of the human body [13]. 

Chaffin et al. [13] are individually measured the maximum range of motion (ROM: Range Of Motion) for each joint motion of human body. As shown in Fig. 6~8, the results measured by Chaffin et al. are indicated together with the measured results and we analyze the movement of shoulder joint of the right arm. First of all, in the flexion-extension movement, when the handle is rotated clockwise, the upper arm contacts the flank and can no longer extend. When the handle is rotated counterclockwise, the flexion up to 90° increases almost linearly to about 30°and then maintains a constant value for handle angles beyond this. In the adduction-abduction movement, the adduction gradually increases to about 20° until the handle rotation angle is 90° counterclockwise, and when the handle rotation angle becomes greater than 90°, the abduction angle increases rapidly. When the handle is rotated clockwise, the adduction angle is small but continues to increase at a constant rate. It can be seen that the internal rotation-external rotation movement changes symmetrically regardless of the rotation direction of the handle. Internal rotation approaches the maximum range of motion, called the limit of joint motion, when the handle rotation angle reaches 120°. In the movement of elbow joint, flexion-extension has a small displacement of about 10° to 15° with respect to the entire angle of the handle. However, the pronation-supination movement of elbow joint is relatively large and the movement is the largest in the elbow joint. In particular, when the handle is rotated counterclockwise by 90°, the pronation angle approaches the maximum range of motion. When the handle is rotated clockwise, the pronation angle decreases, the handle angle becomes 0 at about 70° and then supination occurs. In flexion-extension of the wrist joint, when the handle is rotated counterclockwise, flexion or extension hardly occurs and when the handle is rotated clockwise, slight extension occurs. In the case of radial deviation-ulnar deviation movement, when the handle is rotated counterclockwise, radial deviation occurs up to the maximum range of motion and when the handle is rotated clockwise, radial deviation-ulnar deviation hardly occurs.

In order to calculate the torque generated at each joint when the steering wheel is rotated using the arm movement information obtained through the motion capture system, the driver-steer system linkage model was constructed as shown in Fig. 4 using multi-body dynamics S/W. The driver's arm shown in Fig. 3. is assumed to have 7 degrees of freedom and the hand is fixed to the handle. The steering system consists of a steering wheel and a steering shaft. The steering shaft is connected to the vehicle body and a universal joint and the steering wheel is connected by a cylindrical joint capable of axial translation and rotation with respect to the steering shaft. A steering reaction force is generated as a function of the rotational displacement. When the handle and the steering shaft are mechanically constrained in this way, the handle can be translated in the longitudinal direction of the steering shaft and the handle and the steering shaft combination can be rotated in two directions by the universal joint connecting the steering shaft and the vehicle. The actual steering system has only one rotational degree of freedom about the steering shaft. The three-dimensional trajectory (position and direction) of the hand measured by the motion capture system has a slight difference compared to the actual trajectory due to errors in measurement and signal processing. However, since the hand must move integrally with the handle, the hand cannot follow the circular motion trajectory of the handle due to an error in the trajectory of the hand. The degree of freedom is added to the handle so that the handle can mechanically accommodate the error of the hand movement trajectory. Even if the handle has surplus degrees of freedom, the movement of the handle is driven by the hand movement measured by the motion capture system, so there is no big difference from the actual movement.

A constant torque of 70N-m was generated from the servomotor connected to the steering shaft, and the experimenter was asked to turn the handle as slowly as possible assuming a static state. The driver's posture refers to the SAE standard [14], and the relative position and angle of the steering wheel and the steering factor are determined by the steering angle (θ), the horizontal position of the steering wheel (L), the vertical position of the steering wheel (H) and the upper body inclination angle (ψ). With reference to the driving posture of the experimenter, the values ​​of the design variables are set as shown in Table 1. The joint torque generated in the shoulder, elbow, and wrist joints calculated using the driver-steer system model is shown in Fig. 9 ~ Fig. 15. In the analysis results, it can be seen that the joint torque has a close relationship with the steering wheel rotation angle. It shows that the direction in which the joint torque acts is different depending on the angle of rotation of the handle and the action of the joint torque is switched. The maximum value of the torque acting on each joint is about 40N-m, and the maximum value occurs in the neutral position when the handle is not turned (flexion-extension of the shoulder, external rotation-induction-abduction; pronation-supination of the elbow) or at a certain rotation angle of the handle.

Table 1: Steering wheel position parameters

Figure 5: A measured and simulated steering torque

Figure 6: Shoulder joint rotation angles

Figure 7: Elbow joint rotation angles

Figure 8: Wrist joint rotation angles              

Figure 9: Joint torque of shoulder flexion-extension

Figure 10: Joint torque of shoulder external- internal rotation  

Figure 11: Joint torque of shoulder adduction- abduction

Figure 12: Joint torque of elbow flexion-extension        

Figure 13: Joint torque of elbow pronation supination

Figure 14: Joint torque of wrist flexion-extension   

Figure 15: Joint torque of wrist radial-ulnar deviation

In the autonomous driving environments, key element technologies for analyzing and estimating the occupant's condition based on image-based behavioral data of the occupant were defined. Specifically, technology for estimating the occupant's gaze and extracting feature information on eye movements, facial and head movements, the upper and lower body (center of the upper body) attitude estimation technology of the occupant is defined as the main element technology. Scenarios and tasks for collecting passenger behavior data include not only events related to autonomous driving, but also scenarios of various specific purpose activities of passengers while driving, that is, actions such as manipulating a vehicle's convenience system. In addition, when a specific response action definition for an event occurring in an autonomous driving situation is defined as manual driving switching or operation of a specific control system, operation information of vehicle interior parts is also collected in the simulator at the same time.

Assuming a situation with various driving behaviors according to the preferences of the passengers of the self-driving car, the intent/unintentional behavior and cognition of the passengers are analyzed according to the changes in the traffic situation that appear at this time. Therefore, based on the scenario in Fig. 16, the simulation environment for this is implemented using the driving simulator shown in Fig. 2. In addition, it is possible to establish a safety evaluation environment to determine the ability of the occupants to respond to various unexpected situations (Take over Request, avoiding the accident site, rapid acceleration and deceleration, etc.) that may occur during autonomous driving shown in Fig. 17. A technology capable of determining the occupant's attention state can be based on quantifying the degree of mental behavior of the occupant (driver-centered) in a specific situation by utilizing eye movement information.

Figure 16: Element technology configuration (draft) and flow for recognizing passenger behavior

Figure 17: Example of simulator S/W Carla driving environment

Through the driver's steering motion analysis, the driver’s behavior was analyzed through the motion of the arm joint and the joint torque of the elbow joint which has a great influence on the steering motion. It can be seen that the movement of the arm joint in the steering motion is large in pronation-supination of the elbow joint, flexion-extension of the wrist joint, adduction-abduction and flexion-extension of the shoulder joint. Regarding the change in the angle of inclination of the handle, the effects of flexion-extension of the shoulder joint, flexion-extension of the elbow joint, and adduction-abduction of the shoulder joint were significant in the order of movement. Regarding the change in the horizontal position of the handle, there were significant changes in elbow joint flexion-extension, shoulder joint flexion-extension, and shoulder joint adduction-abduction. Next, with respect to the vertical position change of the handle, flexion-extension of the elbow joint, flexion-extension of the shoulder joint, and adduction-abduction of the shoulder joint were significant. In flexion-extension of the shoulder joint and adduction-abduction of the shoulder joint, the change in rotation angle was relatively large.

The technology reviewed in this study will be used to analyze the behavior of autonomous vehicle occupants. As a body posture estimation technology, HPE (Human Pose Estimation) technology is a key element technology that greatly affects the condition and behavior analysis of occupants, but estimating a person's body posture with images still seems to be a challenging research topic

 This work was supported by the Technology Innovation Program (20019115) funded by the Ministry of Trade, Industry, & Energy (MOTIE, Korea)