2.1 The MIS procedures

The common MIS has three steps: (1) According to the actual surgical needs, a surgeon makes several small incisions (usually 5–15 mm) and inserts a thin tube called trocar. The trocar is deployed as a means of introduction for laparoscope or laparoscopic instruments, like scissors and graspers, to provide an access port during surgery. (2) Creation of a pneumoperitoneum by inflating the abdomen with carbon dioxide to make a separation between organs and increase the operating space of surgical instruments. (3) The surgeon views the magnified image of the patient's internal organs provided by laparoscope on a video monitor. Using different instruments, the surgeon performs a series of surgical operations in the pneumoperitoneum.

This paper takes laparoscopic cholecystectomy (LC) as an example. The surgeon makes three incisions and inserts trocar. In LC, it is always with the patient in a supine position. Three incisions are arranged in an isosceles triangle for better operating space, as shown in Fig. 1. A laparoscope is placed through a trocar, and specialized instruments are placed through other trocars. By operating the laparoscope and instruments, the surgeon delicately separates the gallbladder from its attachments to the liver and the bile duct and then removes it through one incision.

Figure 1The schematic diagram of surgical incisions.

Download

2.2 Layout design of MIS robotic system

The MIS robotic system includes a master-slave manipulator system and a depth camera. The slave manipulator consists of one laparoscope arm and two instrument arms. Laparoscope arm is equipped with a laparoscope, and instrument arms are equipped with different laparoscopic instruments. Laparoscope arm and instrument arms are located on both sides of the operating bed. A depth camera is installed above the operating bed for acquiring the position of the incisions and robotic arms, as shown in Fig. 2.

The three arms have the same mechanical structure. Each arm is divided into three parts, the telecentric fixed-point positioning mechanism, the remote center of motion mechanism and the end effector, as shown in Fig. 3. The first part adjusts the spatial position of telecentric fixed-point by three revolving joints and one linear joint. The second part adjusts the position and posture of the end effector by the master manipulator operated by a surgeon; at its end, there is a versatile quick-change mechanism for end effectors installation.

Figure 2The MIS robotic system.

Download

Figure 3The structure of the robotic arm.

Download

3.1 The mathematical model of artificial pneumoperitoneum

The mathematical model of pneumoperitoneum is established before preoperative planning. The shape of artificial pneumoperitoneum is approximately ellipsoid (Mulier et al., 2008; Oda et al., 2012), so the abdominal wall is simplified to ellipsoid, defined as Eq. (1). The artificial pneumoperitoneal coordinate frame is established by combining the patient's CT images and anatomy. According to anatomy, there are three principal planes, namely the sagittal plane, the coronal plane, and the transverse plane. In the coordinate frame, there are also three reference planes, namely A plane, B plane, and C plane. A plane coincides with the sagittal plane; B plane coincides with the coronal plane; C plane is parallel to the transverse plane, and the pneumoperitoneum is divided equally by C plane. The origin of the coordinate frame is at the intersection of three reference planes. x_p-axis is defined along the mediolateral direction; y_p-axis is defined along the superior-inferior direction; z_p-axis is defined along the anteroposterior direction. In the coordinate frame, the mathematical model of artificial pneumoperitoneum is established, as shown in Fig. 4.

During actual operation, the model parameters (a_p, b_p, c_p) are determined by the medical image and gas insufflation volume. Suppose an adult's chest width is 3.15 dm, chest thickness is 2.45 dm, and chest length is 2.9 dm. The corresponding parameters in Fig. 6 are a_p=1.55, b_p=1.45, h=1.2 dm. Chen suggested that the gas insufflation volume is about 3L (Chen, 1999). According to Eq. (2), calculate c_p=2.27 dm.

Figure 4The coordinate frame and pneumoperitoneum model.

Download

3.2 The lesion parametrization model

The surgeon should be clear about the information of the surgical site, including lesion location, lesion anatomy, and surrounding tissues. At present, the conventional method is imaging (radiology) test, and the lesion model and its surrounding environment are obtained by the 3-D reconstruction technology. Describe the relationship between lesion and incision in parametric form, as shown in Fig. 5. Plane τ represents the target operation plane, a represents the normal vector of the plane τ, d represents the distance from the lesion to the laparoscope, β represents the angle between laparoscope visual axis and a, γ represents the laparoscope deviation angle. So, the two principles of laparoscope arm preoperative planning can be expressed as follows: (1) Observation distance principle: laparoscope-to-target distance d=75–150 mm, d≤ the maximum joint variable of d₇ (definition in Fig. 3), and no barrier (Hanna et al., 1997b). (2) Observation direction principle: axis-to-target view angle, the smaller β is, the better operative field is. When β=0, the operative field is optimum; in other words, the laparoscope visual axis is perpendicular to the plane τ (Hanna and Cuschieri, 1999).

Figure 5The definition of lesion parameters.

Download

3.3 The preoperative planning algorithm framework

Through the study of the mathematical model of artificial pneumoperitoneum, lesion parametrization model and preoperative planning principles, the laparoscope arm preoperative planning algorithm is proposed, as shown in Fig. 6, that includes three stages: data processing and modeling, optimum incision determination and optimum angle determination.

In the first stage, obtain patient information from the medical images, and then establish the mathematical model of artificial pneumoperitoneum, and lastly determine the location and lesion parametrization model. This stage is the basis of the entire algorithm, and also the most time-consuming stage.

In the second stage, all allowable surgical incisions are obtained from the first stage, and then the candidate incisions are determined according to the two principles. The candidate base positions are obtained by the candidate incisions. According to the actual situation of the operating room, select one of the positions as the base position. Combine candidate incisions and the base position to determine the optimum incision.

In the third stage, the candidate entry angles are determined by combining the optimum incision, lesion location, and initial entry angle. Determine the optimum angle according to the observation direction principle. Since there may be no direction in which the visual axis is perpendicular to the plane τ, the minimum β is chosen as the optimum angle.

Finally, the laparoscope arm preoperative planning algorithm is completed, including the optimum incision and the optimum angle.

Figure 6Flow chart of the laparoscope arm preoperative planning algorithm.

Download

3.4 The candidate incisions

The telecentric fixed-point positioning mechanism has four degrees of freedom; the mechanism diagram is shown in Fig. 7. o₄ is the telecentric fixed-point, o₅ is the end of a laparoscope, and α is determined by the remote center of motion mechanism. The prismatic joint is used to adjust the vertical position of o₄, and the three revolute joints are used to adjust the horizontal position. Removing the prismatic joint, it is a planar redundant mechanism. When o₄ remains unchanged, the motion trajectory of the laparoscope is a right circular cone with specific apex at o₄ and aperture π−2α.

Figure 7The mechanism diagram of the telecentric fixed-point positioning mechanism.

Download

First, candidate incisions are determined based on the distance principle. Assume that the positions of the candidate incision and the lesion are P_i and P_l, respectively. If $d\left(P_{\mathrm{i}}{P}_{\mathrm{l}}\right)=|P_{\mathrm{i}}-P_{\mathrm{l}}|\le d_{\mathrm{7}\max }$, P_i satisfies the distance principle. Second, based on the direction principle, the candidate incisions are located on the generatrix of a right circular cone with specific apex at P_l and aperture π−2α. So, candidate incisions are incisions that satisfy the two principles. The following is a mathematical derivation of candidate incisions.

The P_l (x_l, y_l, z_l) is obtained by imaging test; the candidate incisions are located on the intersection (red, Eq. 3) of the abdominal wall (navy blue) and right circular cone with specific apex at P_l (light blue), as shown in Fig. 8. The intersecting line is not a plane curve; it is projected to the plane x_po_py_p for the convenience of research. The projection curve (Eq. 4) is an ellipse whose expression can be obtained by fitting four points on it. Go through P_l and make two planes parallel to y_po_pz_p and x_po_pz_p, point m₁, m₂, m₃ and m₄ are obtained, go through $P_{\mathrm{l}}^{z\mathrm{5}}$ (x_l, y_l, z₅) and make one plane parallel to x_po_py_p, point m₅ and m₆ are obtained, as shown in Fig. 9 and Eqs. (5)–(7). The equation's coefficients can be obtained from any four points in the above six points, and the remaining two points are used to verify the correctness of them.

Figure 8The schematic diagram of candidate incisions.

Download

Figure 9The projection of three planes.

Download

3.5 The candidate base positions

Besides, the base position also affects the surgical incisions. Removing the prismatic joint, the telecentric fixed-point positioning mechanism is a 3-RRR planar redundant mechanism. When o₄ remains unchanged, it is simplified as a planar four-bar mechanism, as shown in Fig. 10. In this case, the link length relationship determines whether the laparoscope trajectory is a whole cone, which makes it possible to provide the optimum operative field. In other words, o₃o₄ should be rotated around o₄ while o₄ is unchanged, that is, the o₃o₄ is a crank. Based on the conditions of crank existence, the link length relationship is determined. Assume that the length of link o₁o₂, o₂o₃, o₃o₄ and o₄o₁ are a₂, a₃, a₄ and l, l determines if there is a crank, which is discussed under three cases, as shown in Eqs. (8)–(10). In summary, the distance from base to fixed-point should be less than $a_{\mathrm{2}}+a_{\mathrm{3}}-a_{\mathrm{4}}$ to ensure that laparoscope has a complete operative field.

1. The link o₁o₄ is the longest link:

   $$\begin{array}{}\text{(8)}& \left\{\begin{array}{l}l\mathit{⩾}{a}_{\mathrm{2}}\\ l+a_{\mathrm{4}}\mathit{⩽}{a}_{\mathrm{2}}+a_{\mathrm{3}}\end{array}\right.\Rightarrow a_{\mathrm{2}}\mathit{⩽}l\mathit{⩽}{a}_{\mathrm{2}}+a_{\mathrm{3}}-a_{\mathrm{4}}\end{array}$$

2. The link o₁o₄ is the shortest link:

   $$\begin{array}{}\text{(9)}& \left\{\begin{array}{l}\mathrm{0}<l\mathit{⩽}{a}_{\mathrm{4}}\\ l+a_{\mathrm{2}}\mathit{⩽}{a}_{\mathrm{3}}+a_{\mathrm{4}}\end{array}\right.\Rightarrow \mathrm{0}<l\mathit{⩽}min(a_{\mathrm{4}},a_{\mathrm{3}}+a_{\mathrm{4}}-a_{\mathrm{2}})\end{array}$$

3. The link o₁o₄ is neither the longest nor the shortest link:

   $$\begin{array}{}\text{(10)}& \left\{\begin{array}{l}{a}_{\mathrm{4}}<l<a_{\mathrm{2}}\\ a_{\mathrm{2}}+a_{\mathrm{4}}\mathit{⩽}l+a_{\mathrm{3}}\end{array}\right.\Rightarrow max(a_{\mathrm{4}},a_{\mathrm{2}}+a_{\mathrm{4}}-a_{\mathrm{3}})<l<a_{\mathrm{2}}\end{array}$$

Figure 10The schematic diagram of the simplified mechanism.

Download

According to Sect. 3.4, the allowable base range is an ellipse. Compared with the projection of the candidate incisions on the plane x_po_py_p, the center coordinate is unchanged and the semi-major axis and semi-minor axis increase l_max. The intersection of allowable base range and non-interference area in the operating room is the candidate base positions, as shown in Eq. (6).

3.6 The optimum incision and the optimum angle

P_b (x_b, y_b, z_b) is chosen as the base position, so the optimum incisions are within allowable incisions circle with the P_b as the origin and l_max as the radius. The optimum incisions (red) are located on the intersection of candidate incisions (green) and allowable incisions circle (black), as shown in Fig. 11 and Eq. (7). The optimum angle of laparoscope entry is β=0, that is, laparoscope visual line coincides with the line relating incision to the lesion. To sum up, combined with Sects. 3.5 and 3.6, the laparoscope arm preoperative planning algorithm is completed.

Given a set of parameters based on the actual situation, the steps of the algorithm are described in detail, a₂=220 mm, a₃=220 mm, a₄=150 mm, α=45^∘, $P_{\mathrm{l}}=(\mathrm{0.35},\mathrm{0.2},\mathrm{0.3})$, $P_{\mathrm{l}}^{z\mathrm{5}}=(\mathrm{0.35},\mathrm{0.2},\mathrm{1.5})$ (in the o_p−x_py_pz_p coordinate frame).

3.6.2 Step 2 Determine base position

The candidate base positions are located on the curve, as shown in Eq. (8). Within the allowable base range, choose a base position $P_{\mathrm{b}}=(\mathrm{2.3},-\mathrm{0.2}$, z_b), z_b is determined according to the condition of the operating room.

$$\begin{array}{}\text{(15)}& {\displaystyle }\left\{\begin{array}{l}{\displaystyle \frac{{\left(x-\mathrm{0.13}\right)}^{\mathrm{2}}}{(\mathrm{1.16}+\mathrm{2.9}{)}^{\mathrm{2}}}}+{\displaystyle \frac{{\left(y-\mathrm{0.07}\right)}^{\mathrm{2}}}{(\mathrm{1.11}+\mathrm{2.9}{)}^{\mathrm{2}}}}\mathit{⩽}\mathrm{1}\\ x>\mathrm{1.55}\end{array}\right.\end{array}$$

Figure 11The schematic diagram of optimum incisions.

Download

3.6.3 Step 3 Determine the optimum incision and the optimum angle

First, determine x_i based on the surgical needs, body condition and surgeon's operating habits, calculate y_i based on Eq. (9), calculate z_i based on the mathematical model of pneumoperitoneum, the optimum incision is (x_i, y_i, z_i). Second, the optimum visual axis direction is the line connecting the optimum incision to the lesion. The optimum incision and optimum angle are shown in Fig. 12.

$$\begin{array}{}\text{(16)}& {\displaystyle }\left\{\begin{array}{l}{\displaystyle \frac{{\left(x-\mathrm{0.13}\right)}^{\mathrm{2}}}{\mathrm{1.34}}}+{\displaystyle \frac{{\left(y-\mathrm{0.07}\right)}^{\mathrm{2}}}{\mathrm{1.23}}}=\mathrm{1}\\ (x-\mathrm{2.3}{)}^{\mathrm{2}}+(y+\mathrm{0.2}{)}^{\mathrm{2}}<{\mathrm{2.9}}^{\mathrm{2}}\end{array}\right.\end{array}$$

Figure 12The sketch map of the optimum incision and angle.

Download

4.1 Problem description

Reinforcement learning describes the set of learning problems where an agent should learn how to map states to actions in an environment to maximize the defined reward function. Throughout the learning process, an agent is not told which actions to take but instead should find out which action yield the most reward by trying various actions. In most cases, actions may affect not only the immediate reward but also the next state, and through that all subsequent rewards. In solving practical problems, it should define a reasonable reward function to compute the reward for taking actions and have a goal relating to the state of the environment. Also, it should quantify all the variables the environment describes and have access to these variables at each step or state.

In this paper, the agent is the 3-RRR planar redundant mechanism which is a simplified model of telecentric fixed-point positioning mechanism plus laparoscope. The environment is the lesion and the surgical incision obtained through the preoperative planning algorithm. The actions are the movement of three revolute joints. The agent–environment interaction is shown in Fig. 13.

Figure 13The agent–environment interaction in reinforcement learning.

Download

4.2 Deep deterministic policy gradient (DDPG)

In this paper, laparoscope arm automatic positioning is achieved by DDPG, which is a model-free, off-policy actor-critic algorithm based on the deterministic policy gradient (DPG) (Silver et al., 2014). Deep neural network (DNN) function approximators were used to estimate the action-value function. Thus, the algorithm can learn policies in high-dimensional, continuous action spaces.

Based on DPG, DDPG combines the ideas underlying the success of Deep Q Network (DQN) (Mnih et al., 2013, 2015). It can learn value functions stably and robustly due to two aspects. First, the network is trained off-policy with samples from a replay buffer to minimize correlations between samples. Second, the network is trained with a target Q network to give consistent targets during temporal difference backups. Meanwhile, batch normalization is used to accelerate deep network training and improve the accuracy of the model (Ioffe and Szegedy, 2015).

DDPG contains a parameterized actor function μ(s|θ^μ) and critic network Q(s, a|θ^Q) with weights θ^μ and θ^Q. The critic network is learned using the Bellman equation (Eqs. 10–10) to make the L(θ^Q) smaller and smaller. In other words, Q(s, a|θ^Q) gets closer to the actual value.

where

The actor function is updated by the chain rule (Eq. 11) to the expected return from the start distribution J with respect to the actor parameters.

Every n steps DDPG updates the target networks of actor and critic using “soft” target updates (Eq. 12), rather than directly copying the weights.

4.3 Reward function construction

In the training process, telecentric fixed-point (marked point) position and lesion location are taken as the input of the DDPG algorithm. The fixed-point is obtained by a depth camera, the optimum incision, the optimum angle and the base position are obtained by the preoperative planning algorithm. The DDPG algorithm that combines the algorithm can learn policies directly from the inputs, to achieve laparoscope arm automatic positioning for the robot-assisted laparoscopic surgery. The reward function is essential for the algorithm to learn policies successfully. It consists of intermediate reward and final reward, where the former is given a continuous, guided negative reward when the task is not completed, and the latter is given a positive reward that is one to two orders of magnitude larger than the former when the task is completed. The continuous reward function can make convergence of the algorithm better.

In the o_p−x_py_pz_p coordinate frame, the fixed-point position is P_f (x_f, y_f, z_f), the incision position is P_i (x_i, y_i, z_i), the laparoscope end position is P_e (x_e, y_e, z_e), and the lesion location is P_l (x_l, y_l, z_l). The goal of the task is $\left|P_{\mathrm{f}}{P}_{\mathrm{i}}\right|+\left|P_{\mathrm{e}}{P}_{\mathrm{l}}\right|=$0 (lsinα (definition in Fig. 7) is equal to $\left|P_{\mathrm{i}}{P}_{\mathrm{l}}\right|$ for programming convenience.). The intermediate reward is $-\left(\right|P_{\mathrm{f}}{P}_{\mathrm{i}}|+|P_{\mathrm{e}}{P}_{\mathrm{l}}|$) and is normalized to [$-\mathrm{1},\mathrm{0}$] interval. The final reward is 10.

4.4 States description

To improve the convergence of the algorithm, the state variables also play a crucial role in addition to the reward function. If state variables can adequately present the environment, the algorithm can learn policies quickly. Because the image from the depth camera contains all the state information of the environment, it is reasonable to use the image directly as input. However, due to the limitations of the hardware, the processing image data is very slow. To speed up training of the algorithm, it uses a low-dimensional states description, such as joint variables and positions, instead of high-dimensional renderings of the environment.

The algorithm is to make the laparoscope arm move to the target position, so the joint variables are used as the state variables. However, from the training results, these variables cannot adequately describe the environment; in other words, the algorithm cannot achieve the laparoscope arm automatic movement. So, the distance from telecentric fixed-point to incision, the distance from laparoscope end to the lesion, and whether the target is reached are added to the state variables. The experimental results of these two state variables are described in Sect. 5.2.

5.1 Simulation details

The environment is simulated using Pyglet, including a lesion point, a surgical incision and a simplified model of the telecentric fixed-point positioning mechanism. For this environment, a lesion point is randomly specified within a reasonable range, an incision and a base location are obtained by the preoperative planning algorithm. Batch normalization is used on the state input, all layers of the actor network and all layers of the critic network before the action input. In this way, it can learn effectively across tasks with different types of units, without needing to ensure the units are within a set range manually.

TensorFlow is used in the code for high-performance numerical computation. The simulations use Adam (Kingma and Ba, 2015) for learning neural network parameters with a learning rate of 10⁻⁵ for the actor and critic. For Q it includes L₁ weight decay of 0.1, L₂ weight decay of 10⁻³ and a discount factor of γ=0.9. For the soft target updates, it uses τ=0.01. The neural networks use the rectified non-linearity for all hidden layers (Glorot et al., 2011). The networks have three hidden layers with 900, 900 and 60 units respectively, and the final output layer of the actor is a tanh layer, to bound the actions. The actions are not included until the 3rd hidden layer of Q. The layers weights and biases of both the actor and critic are initialized from a uniform distribution [−x, x], where x=sqrt (6./(in + out)). It trains with minibatch sizes of 16, and it uses a replay buffer size of 6×10⁴. The behavior policy during training is ε-greedy with ε annealed linearly from 1 to 0.1 over the first hundred episodes and fixed at 0.1 after that. The simulations train for a total of 2000 episodes; every episode is terminated if the goal is not completed after 600 steps.

5.2 Simulation results

Two simulations are set up to evaluate the performance of the improved method applied to laparoscope arm automatic positioning for the robot-assisted laparoscopic surgery. The two simulations make one change to states description during training only, and use the same network architecture, learning algorithm and hyperparameters settings. States descriptor one is three joint variables and states descriptor two is the former plus the distance from fixed-point to incision, the distance from laparoscope end to the lesion, and whether the target is reached.

The two simulations evaluate the policy periodically during training by testing it without exploration noise. The improved method with 3 action dimensions and 20 state dimensions runs ten times in the simulated environment. Performance after training across the environment for at most 2000 episodes. The results of ten training sessions report both total reward per episode and steps to target, as shown in Figs. 14–17. The solid line in the figure represents the average over ten sessions, the upper boundary of the shadow part represents the maximum over ten sessions, and the lower boundary represents the minimum value.

Figure 14The total reward per episode with states descriptor one.

Download

Figure 15The steps to target with states descriptor one.

Download

Figure 16The total reward per episode with states descriptor two.

Download

Figure 17The steps to target with states descriptor two.

Download

Figure 14 shows that the average of total reward per episode is stabilized to negative and only a few episodes total reward are positive. Figure 15 shows that the steps to target are always 600. These two figures show that it never reaches the goal. Figure 16 shows that the average of total reward per episode increases from −300 to about 120. After 400 episodes, the total reward converges to around 120. Figure 17 shows the steps to target stabilizes at about 150. These two figures show that it reaches the goal after 400 episodes. The results illustrate the states descriptor two is outperformed states descriptor one, the latter does not enable the agent to converge to a good solution, but the former can do it. In other words, the improved method which uses the states descriptor two can learn the right policies on laparoscope arm automatic positioning.