2.1. The DNS of the turbulent channel flows

We consider the turbulent channel flows established between two parallel plates separated by $2h$ , driven by a pressure gradient. The governing equations of the turbulent flow are the Navier–Stokes equations of an incompressible Newtonian fluid, written as

(2.1) \begin{align} \frac {\partial u_{j}}{\partial x_{j}} = 0, \end{align}

(2.2) \begin{align} \frac {\partial u_{i}}{\partial t}+u_{j}\,\frac {\partial u_{i}}{\partial x_{j}} =-\frac {1}{\rho }\,\frac {\partial p}{\partial x_{i}}+\nu\, \frac {\partial ^{2}u_{i}}{\partial x_{j}\,\partial x_{j}}+f_{1}\delta _{i1}, \end{align}

where $x_{i}(i=1,2,3)=(x,y,z)$ represents the coordinates in the streamwise, wall-normal and spanwise directions, respectively, and $u_{i}(i=1,2,3)=(u,v,w)$ denotes the corresponding velocity components. Here, $t$ is the time, $\rho$ is the density, $p$ is the pressure, and $\nu$ is the kinematic viscosity. A body force $f_{1}$ is introduced to maintain constant momentum in the channel, ensuring the averaged bulk velocity $U_m$ in the channel flow.

The flow is assumed to be periodic in the streamwise and spanwise directions, with periods $L_x$ and $L_z$ , respectively. The upper wall imposes no-slip and no-penetration conditions, setting the velocities $u=v=w=0$ . On the other hand, the lower wall adheres to the no-slip condition with $u=w=0$ , and implements turbulent control through blowing and suction.

The code AFiD (Verzicco & Orlandi Reference Verzicco and Orlandi1996; van der Poel et al. Reference van der Poel, Erwin, Ostilla-Mónico, Donners and Verzicco2015; Zhu et al. Reference Zhu2018) was utilized to carried out the DNS of turbulent channel flows. An energy-conserving second-order finite difference scheme is applied in the spatial discretization, with velocities on a staggered grid. Time marching is performed using a third-order Runge–Kutta scheme, combined with a Crank–Nicolson scheme for the implicit terms. The grids are uniformly distributed in both the streamwise and spanwise directions, with wall-normal grid refinement applied near the walls.

The computational parameters are listed in table 1 for the three Reynolds numbers $Re=U_{m}h/\nu$ considered in this study. The friction velocity $u_{\tau }=\sqrt {\tau _{w}/\rho }$ and the friction Reynolds number $Re_{\tau }=h^{+}=u_{\tau }h/\nu$ define wall units in the following discussions, denoted by a + superscript, where $\tau _{w}$ is the skin friction. Here, $y^{+}=y/\delta _{\nu }$ , where $\delta _{\nu }=\nu /u_{\tau }=h/Re_{\tau }$ is the friction length.

Table 1. Computational parameters. Here, $\Delta _x$ , $\Delta _y$ and $\Delta _z$ are the resolutions in the streamwise, wall-normal and spanwise directions, respectively.

2.2. The DRL methodology

Figure 1. The flow chart of reinforcement-learning-driven control in turbulent channel flows.

In order to control the turbulent channel flows, blowing and suction based on the DRL, predictions are applied to the lower wall. The flow chart of the control driven by reinforcement learning is shown in figure 1. Our current program mainly consists of two parts: the numerical simulation part and the reinforcement learning part. The numerical simulation part, as discussed in § 2.1, acts as the environment and outputs the state $s_{t}$ and reward $r_{t}$ obtained in the flow field. The reinforcement learning part, acting as the agent, receives these variables, optimizes the decision-making policy $\pi (s_{t})$ based on the reward, and outputs actions $a_{t}$ based on the state. The numerical simulation part then uses these actions to control the flow, and advances the simulation in time. This creates a loop to achieve active control driven by reinforcement learning. Here, we select the wall blowing and suction velocities $v_{w}^{\prime }$ as the actions, and we choose the streamwise velocity fluctuations $u^{\prime }(x,z)\mid _{y^{+}=15}$ in the near-wall region as the states, similar to those adopted by Sonoda et al. (Reference Sonoda, Liu, Itoh and Hasegawa2023). Velocity fluctuations are defined based on the mean velocity profile of each case, where $u^{\prime }=u(x,y,z)-U(y)$ . The mean wall blowing and suction velocity is set to zero.

The agent that we adopted is based on the open-source code provided by Lee et al. (Reference Lee, Kim and Lee2023), which employs the twin-delayed deep deterministic policy gradient (TD3) model (Lillicrap et al. Reference Lillicrap, Hunt, Pritzel, Heess, Erez, Tassa, Silver and Wierstra2015), an actor–critic network structure. The TD3 model offers improved stability and performance in learning by addressing overestimation bias, incorporating delayed updates, and implementing target smoothing. It has been proven to be suitable for turbulence control optimization (Lee et al. Reference Lee, Kim and Lee2023). In the TD3 model, the goal is to optimize the action value function $q_{\pi }(s_{t},a_{t})$ by satisfying the Bellman equation, where

(2.3) \begin{equation} q_{\pi }(s_{t},a_{t})=\mathbb {E}\big[r_{t}^{d}+\gamma ^{n}q_{\pi }(s_{t+n},\pi _{\phi }(s_{t+n})+\epsilon )\big]. \end{equation}

Here, $r_{t}^{d}=\sum _{j=1}^{n}\gamma ^{j-1}r_{t+j}$ is the $n$ -step reward, $\gamma$ is the discounted factor, $\pi _{\phi }(s_{t+n})$ is the delayed policy update, and $\epsilon$ is the clipped random noise. We adopt $n=5$ and $\gamma =0.95$ in all cases, following Lee et al. (Reference Lee, Kim and Lee2023). The expected cumulative reward is predicted by the critic networks, and the objective function for updating the parameters of the critic networks is given by

(2.4) \begin{equation} J(\theta )=N^{-1}\sum \big[r_{t}^{d}+\gamma ^{n}q_{\pi }(s_{t+n},\pi _{\phi }(s_{t+n})+\epsilon )-q(s_{t},a_{t})\big]^{2}, \end{equation}

where $N=64$ is the minibatch size, and $\theta$ is the weight parameter of the critic networks. The actor network aims to find an optimal policy, guided by the policy objective function $J(\phi )$ . Here, $\phi$ represents the weight parameters of the actor (or critic) networks. The objective function is updated by

(2.5) \begin{equation} \boldsymbol{\nabla}_{\psi }J(\psi )=\mathbb {E}\big[\boldsymbol{\nabla}_{a}q_{\pi }(s_{t},a_{t})\mid _{a=\pi (s_{t})}\boldsymbol{\nabla}_{\psi }\pi _{\psi }(s_{t})\big], \end{equation}

where $\psi$ is the weight parameter of the actor network. The actor network includes three convolutional layers, with the first two layers activated by the ReLU function. The numbers of filter kernels for these layers are set to $64$ , $32$ and $1$ , respectively, with each filter kernel sized at $3\times 3$ . In contrast, the critic network is structured with six convolutional layers followed by three fully connected layers, all activated by the ReLU function. Each convolutional layer contains $32$ filter kernels of size $3\times 3$ . Additionally, an average pooling layer is applied after every two convolutional layers. The fully connected layers each consist of $32$ neurons, and the network ultimately outputs a $q$ value to evaluate the control policy. Detailed hyperparameters can be referred to in Lee et al. (Reference Lee, Kim and Lee2023).

The choice of reward $r_{t}$ is crucial for the effectiveness of the training outcomes. Inspired by the optimal control (Bewley, Moin & Temam Reference Bewley, Moin and Temam2001), we define the reward $r=1-e/e_{0}$ as the reduction of integrated turbulent kinetic energy (TKE) in the lower half-channel at the end of each state step. Here, $e$ is the integrated TKE with control, defined as

(2.6) \begin{equation} e=\frac {1}{L_{x}L_{z}hU_{m}^{2}}\int _{0}^{L_{z}}\int _{0}^{h}\int _{0}^{L_{x}}\left (\frac {1}{2}\,u_{i}^{\prime }u_{i}^{\prime }\right )\mathrm {d}x\,\mathrm {d}y\,\mathrm {d}z, \end{equation}

and $e_0$ is the integrated TKE without control. It is important to note that the reward is used only during the model training process, and is no longer needed once the model has converged.

Furthermore, the time lengths of each state step and episode for all cases are shown in table 2. First, $\Delta t$ needs to be sufficiently long to allow the changes in the control strategy to fully develop. Therefore, $\Delta t^{+}\approx 50$ is selected, which also meets the requirement for the prediction horizon $\Delta t^{+}\gt 25$ in optimal control (Bewley et al. Reference Bewley, Moin and Temam2001). Additionally, given that the maximum ${Re}_{\tau }$ in our cases reaches $1000$ , it is crucial to train a control strategy that remains effective under large-scale structure evolution. Consequently, we ensure that $\Delta T$ is at least $20h/U_{m}$ , which exceeds the time required for large-scale structures to advect streamwise across the entire channel, approximately $L_x/U_m\thickapprox 6h/U_{m}$ .

Table 2. Parameters of state steps and episodes. Here, $\Delta t$ and $\Delta T$ are the time lengths of each state step and episode, respectively, while $N_{st}$ is the number of state steps in one episode.

In summary, a comparison of our computational method with previous DRL-based studies on turbulent channel control is shown in table 3. Our work uses the TD3 algorithm, similar to that of Lee et al. (Reference Lee, Kim and Lee2023), with streamwise velocity fluctuations $u^{\prime }$ as input states, and the TKE reduction rate as the reward. We have extended DRL-based wall blowing and suction control to higher Reynolds numbers, reaching ${Re}_{\tau }=1000$ .

Table 3. Comparison of computational details in DRL-based turbulent channel control studies. Here, DDPG denotes the deep deterministic policy gradient algorithm. In all the studies, the output actions selected are the wall blowing and suction velocities $v_{w}^{\prime }$ .

Table 4. The DNS cases and drag reduction results. Here, $DR$ represents the drag reduction rate, $P_{S}/P_{I}$ denotes the power saving ratio (where $P_{S}$ is the power saving, and $P_{I}$ is the power input), $\Delta U_{s}^{+}$ denotes the shift of the mean velocity profile in the logarithmic region, $y_{vw}$ indicates the height of the virtual wall, and $-\langle u^{\prime }v^{\prime }\rangle _{vw}$ is the averaged residual Reynolds stress on the virtual wall.

3.1. Performance of the DRL models

In this study, we focus on the DRL-optimized control models under different blowing and suction intensities, and their impact on the flow mechanism. The DNS cases that we utilized are detailed in table 4. Among these, cases with the suffices 0 and ‘opp’ did not use DRL models. The former denotes cases with no blowing or suction, whereas the latter represents cases with opposition control as suggested by Choi et al. (Reference Choi, Moin and Kim1994). Cases with suffices 1, 2 and 3 are based on DRL-optimized control models, where the magnitude of wall blowing and suction $v_{w}^{\prime }$ is limited to $ [-u_{\tau }^{0},u_{\tau }^{0} ]$ , $ [-2u_{\tau }^{0},2u_{\tau }^{0} ]$ and $ [-3u_{\tau }^{0},3u_{\tau }^{0} ]$ , respectively. Here, the superscript 0 denotes variables before the application of turbulence control.

Figure 2. The evolution of the normalized reward over episodes during the training process: (a) C180, (b) C550, (c) C1000. denotes cases with suffix 1; denotes cases with suffix 2; denotes cases with suffix 3.

Before further analysis, it is essential to confirm the training status of the current DRL models. The normalized reward $\overline {r}$ , defined as $\overline {r}=\sum _{j=1}^{n}\gamma ^{j-1}r_{t+j}/\sum _{j=1}^{n}\gamma ^{j-1}$ , serves as an indicator of learning performance during the training of the control strategy. The evolution of $\overline {r}$ over episodes for different cases is illustrated in figure 2. Significant oscillations are observed primarily within the first $10$ episodes, while the rewards for all analysed cases gradually converge after $10$ episodes, indicating stabilization of the DRL models. Consequently, the model at $20$ episodes will be utilized uniformly as the control strategy for subsequent analysis. Additionally, we observed that further training beyond $20$ episodes does not significantly enhance learning performance, although this is not shown in figure 2. As suggested by Lee et al. (Reference Lee, Kim and Lee2023), prolonged training can lead to issues such as catastrophic forgetting or overfitting, potentially causing the training process to fail. Therefore, the models at $20$ episodes are deemed appropriate for our purposes.

The drag reduction effects under different blowing and suction intensities are shown in table 4. As the range of $v_{w}^{\prime }$ is expanded, the drag reduction rates ( $DR$ ) achieved using DRL models continuously improve. When $v_{w}^{\prime }\in [-3u_{\tau },3u_{\tau } ]$ , the drag reduction effect significantly surpasses that of the traditional opposition control method, including high-Reynolds-number cases. Specifically, the drag reduction rate is $35.6\,\%$ in case C180-3, $30.4\,\%$ in case C550-3, and $27.7\,\%$ in case C1000-3. On the other hand, it is noted that the maximum drag reduction rate achieved using the DRL model decreases as the Reynolds number increases. This trend is similar to that observed with the opposition control method (Chang et al. Reference Chang, Collis and Ramakrishnan2002; Iwamoto et al. Reference Iwamoto, Suzuki and Kasagi2002; Pamiès et al. Reference Pamiès, Garnier, Merlen and Sagaut2007; Touber & Leschziner Reference Touber and Leschziner2012; Hwang Reference Hwang2013). We also tested several cases with modified input states or rewards, along with the performance of trained models under different resolutions and $Re_{\tau }$ ; see Appendix A.

Furthermore, as proposed by Bewley et al. (Reference Bewley, Moin and Temam2001), the energy efficiency of the active control policy can be quantified by the ratio of power saving to power input, $P_{S}/P_{I}$ , as shown in table 4. The power input $P_{I}$ is calculated as

(3.1) \begin{equation} P_{I}=\left \langle \left |p_{w}^{\prime }v_{w}\right |\right \rangle +\rho \big \langle \big |v_{w}^{3}\big|\big\rangle /2, \end{equation}

where $p_{w}^{\prime }$ represents pressure fluctuations on the wall, and $\rho$ denotes the fluid density. The power saving $P_{S}$ , in turn, is given by $P_{S} = (\tau _{w}^{0} - \tau _{w}) U_{m}$ , where $\tau _{w}^{0}$ is the wall shear stress before control. For cases with ${Re}_{\tau } \leq 550$ , the power saving ratio $P_{S}/P_{I}$ shows an increasing trend as the range of $v_{w}^{\prime }$ expands. Specifically, at ${Re}_{\tau } = 180$ , the power saving ratio achieved by DRL-based control is lower than that achieved by opposition control. On the other hand, at ${Re}_{\tau } = 550$ , DRL-based control demonstrates more effective energy savings compared to opposition control. As the Reynolds number increases, the power efficiency ratio $P_{S}/P_{I}$ gradually declines, eventually dropping to approximately $5$ at ${Re}_{\tau } = 1000$ . Notably, at ${Re}_{\tau } = 1000$ , adjustments in the range of $v_{w}^{\prime }$ have only a minor impact on the energy savings achieved by DRL-based control.

3.2. Velocity statistics

Further investigation is needed to understand the impact of DRL-based control on flow field statistics and the underlying mechanisms affecting the flow. Therefore, this subsection will compare the velocity statistics across different cases.

Figure 3. Mean velocity profile under different control strategies: (a) C180, (b) C550, (c) C1000. denotes cases with suffix 0; denotes cases with suffix 1; denotes cases with suffix 2; denotes cases with suffix 3.

Figure 3 shows the wall-normal distributions of the mean velocity profile, $U^{+}$ . In the viscous sublayer below $y^+=5$ , the wall blowing and suction based on the DRL model result in a decrease in the mean velocity compared to the uncontrolled case. In the logarithmic region, the velocity profiles continue to follow the logarithmic law even in the presence of control, but with an upward shift relative to the uncontrolled case. This behaviour is analogous to what is observed with opposition control (Choi et al. Reference Choi, Moin and Kim1994). The profile shift $\Delta U_{s}$ is detailed in table 4, with $\Delta U_{s}$ calculated as the averaged vertical shift between $y^+ = 50$ and $y/h = 0.5$ . As the range of $v_{w}^{\prime }$ is extended progressively, the drag reduction rate increases consistently, resulting in a corresponding rise in the mean velocity profile in the logarithmic region, and a gradual increase in $\Delta U_{s}$ . Furthermore, $\Delta U_{s}$ at higher Reynolds numbers gradually decreases, corresponding to a decline in the drag reduction rate.

Figure 4. Wall-normal distributions of the velocity fluctuations under different control strategies. Solid, dashed and dash-dotted lines represent $u_{rms}^{\prime }$ , $v_{rms}^{\prime }$ and $w_{rms}^{\prime }$ , respectively: (a) C180, (b) C550, (c) C1000. denotes cases with suffix 0; denotes cases with suffix 1; denotes cases with suffix 2; denotes cases with suffix 3.

Wall-normal distributions of velocity fluctuations under different control strategies are illustrated in figure 4. After applying control, the streamwise velocity fluctuations $u_{rms}^{\prime }$ , indicated by solid lines, show a significant increase in the viscous sublayer below $y^+=5$ . In contrast, at higher positions, particularly around $y^+=10$ in the near-wall region, the peak of $u_{rms}^{\prime }$ vanishes. The reduction in streamwise velocity fluctuations becomes more pronounced as the range of blowing and suction velocities is further expanded. This trend is observed consistently across different Reynolds numbers. However, it is noteworthy that for $Re_{\tau }^{0}\thickapprox 550$ and $1000$ , the impact of wall blowing and suction is trivial in the outer region, as depicted in figures 4(b) and 4(c). The current DRL-based control strategy hardly affects the outer region, which is dominated by large-scale structures. On the other hand, the application of control significantly increases the wall-normal velocity fluctuations $v_{rms}^{\prime }$ within the viscous sublayer, due to the direct impact of blowing and suction. Conversely, the spanwise velocity fluctuations $w_{rms}^{\prime }$ below $y^+=5$ exhibit minimal changes. As the range of blowing and suction velocities is further extended, both $v_{rms}^{\prime }$ and $w_{rms}^{\prime }$ within $10\lt y^{+}\lt 30$ gradually decrease. This trend is particularly evident at the low Reynolds number $Re_{\tau }^{0}\thickapprox 180$ , as shown in figure 4(a), but becomes less pronounced at higher Reynolds numbers. Additionally, in the controlled cases, $v_{rms}^{\prime }$ in the near-wall region initially decreases and then increases with height. The point of minimum $v_{rms}^{\prime }$ in the near-wall region can be defined as the position of the virtual wall $y_{vw}$ (Hammond et al. Reference Hammond, Bewley and Moin1998). The virtual wall and the residual fluctuations on it will be discussed further in § 3.3.

Figure 5. Wall-normal distributions of the averaged Reynolds shear stress under different control strategies: (a) C180, (b) C550, (c) C1000. denotes cases with suffix 0; denotes cases with suffix 1; denotes cases with suffix 2; denotes cases with suffix 3.

Figure 5 presents the wall-normal distributions of the averaged Reynolds shear stress, where $ \langle \varphi \rangle$ denotes the variable $\varphi (x,y,z,t)$ averaged over the streamwise, spanwise and temporal directions. Although not shown in the figure, the Reynolds stress at the wall is always zero, confined by the boundary conditions. The Reynolds stress in the viscous sublayer is higher in the controlled case compared to the uncontrolled case, due to the application of blowing and suction at the wall. In the controlled case, the Reynolds stress slightly increases with height, reaching a peak near $y^+=5$ , before rapidly decreasing and forming a trough at approximately $y^{+}=10$ – $12$ . Compared to the uncontrolled case, the Reynolds stress with control significantly decreases in the region $10\lt y^{+}\lt 20$ . This decreasing trend becomes more pronounced as the range of $v_{w}^{\prime }$ is further extended. According to the FIK identity proposed by Fukagata, Iwamoto & Kasagi (Reference Fukagata, Iwamoto and Kasagi2002), this reduction in Reynolds stress also leads directly to a decrease in the skin friction. At a low Reynolds number $Re_{\tau }^{0}\thickapprox 180$ , the Reynolds stress in the logarithmic region is lower with control, as shown in figure 5(a). However, this trend gradually disappears at higher Reynolds numbers, corresponding to a decrease in drag reduction rate.

The relationship between wall blowing and suction velocity and the velocity fluctuations at the detection plane is a crucial aspect of flow control. Traditional opposition control employs blowing and suction with equal magnitudes but opposite directions to the wall-normal velocity fluctuations at the detection plane. Consequently, the correlation $R$ between the blowing and suction velocity $v_{w}^{\prime }$ and $v^{\prime }$ at $y^+=15$ , defined as

(3.2) \begin{equation} R\big(v_{w}^{\prime },v^{\prime }\big)=\frac {\big\langle v_{w}^{\prime }v^{\prime }\big\rangle }{\sqrt {\big \langle v_{w}^{\prime }v_{w}^{\prime }\big \rangle \left \langle v^{\prime }v^{\prime }\right \rangle }}, \end{equation}

would be strictly $-1$ . Unlike opposition control, the current DRL-based control strategy is based on the streamwise velocity fluctuations $u^{\prime }$ at the detection plane $y^+=15$ as the input state. The joint probability density function (p.d.f.) of wall blowing and suctions $v_{w}^{\prime }$ with velocity fluctuations at the near-wall detection plane is shown in figure 6, using the results from case C1000-3 as an example. Here, $v_{w}^{\prime }$ has a relatively weak correlation with $v^{\prime }$ at the detection plane $y^+=15$ , as illustrated in figure 6(a), where $R(v_{w}^{\prime },v^{\prime })=-0.10$ . Conversely, the joint p.d.f. between $v_{w}^{\prime }$ and $u^{\prime }$ at $y^+=15$ in figure 6(b) is predominantly aligned with the first and third quadrants. This alignment indicates that wall blowing tends to occur beneath high-speed regions near the wall, while suction is more likely beneath low-speed regions. Furthermore, the correlation $R(v_{w}^{\prime },u^{\prime })=0.71$ is positive. This behaviour closely resembles the mechanism identified by Lee et al. (Reference Lee, Kim and Lee2023), where their DRL models, using wall streamwise shear stress as input states, exhibited a similar trend of applying blowing beneath high-speed regions and suction beneath low-speed streaks. They also observed a strong correlation between wall actuation and streamwise wall shear stress, similar to figure 6(b), suggesting that these DRL models effectively reduce drag through direct control of sweep and ejection events. Despite not being shown in the figure, this pattern appears consistently across all cases in our current work, implying a stronger connection between the DRL-based control strategies and the streamwise velocity fluctuations as the input state.

Figure 6. Joint p.d.f. of the wall blowing and suction $v_{w}^{\prime }$ with (a) $v^{\prime }$ and (b) $u^{\prime }$ at $y^+=15$ in case C1000-3. The white diagonals denote (a) $v^{\prime }=-v_{w}^{\prime }$ and (b) $u^{\prime }=v_{w}^{\prime }$ , respectively. Contour levels are $0.1 (0.1) 0.8$ of the maximum probability density.

3.3. Kinematic analysis of drag reduction based on virtual wall theory

The DRL-based control strategy could lead to larger drag reduction compared to the traditional opposition control; however, the underlying mechanism requires further investigation. This subsection utilizes the virtual wall theory proposed by Hammond et al. (Reference Hammond, Bewley and Moin1998) to analyse the drag reduction mechanism from a kinematic perspective.

According to the virtual wall theory by Hammond et al. (Reference Hammond, Bewley and Moin1998), wall blowing and suction create a virtual wall between the actual wall and the detection plane. This virtual wall hinders streamwise vortices from bringing high-speed fluid to the wall, which would otherwise create local high friction zones, thereby resulting in drag reduction. The drag reduction effect is influenced mainly by two factors: the height of the virtual wall $y_{vw}$ , and the magnitude of the residual Reynolds stress on the virtual wall $- \langle u^{\prime }v^{\prime } \rangle _{vw}$ . Specifically, the higher the virtual wall, the better the drag reduction effect. And the lower the residual Reynolds stress, the stronger the virtual wall’s ability to impede wall-normal momentum transport, resulting in better drag reduction.

The height of the virtual wall and the residual Reynolds stress under different control strategies are detailed in table 4. At the low Reynolds number $Re_{\tau }^{0}\thickapprox 180$ , as the range of blowing and suction velocities is further expanded, the height of the virtual wall gradually increases, and the residual Reynolds stress on the virtual wall gradually decreases. Both these changes correspond to an improvement in drag reduction effect. The values of $y_{vw}$ and $- \langle u^{\prime }v^{\prime } \rangle _{vw}$ for C180-2 and C180-3 are similar, resulting in comparable drag reduction rates for both cases. Compared to the traditional opposition control method, the DRL-based control strategy in C180-3 does not show a significant reduction in residual stress on the virtual wall. However, its primary benefit is the ability to further elevate the virtual wall. As the Reynolds number increases, the height of the virtual wall under the DRL-based control strategy is significantly lower for $Re_{\tau }^{0}\thickapprox 550$ and $1000$ compared to the results for $Re_{\tau }^{0}\thickapprox 180$ . Additionally, the residual stress $- \langle u^{\prime }v^{\prime } \rangle _{vw}$ on the virtual wall rapidly increases with the rising Reynolds number. In case C550-3, $-\langle u^{\prime }v^{\prime } \rangle _{vw}$ is approximately three times that of case C180-3, and in case C1000-3, $- \langle u^{\prime }v^{\prime } \rangle _{vw}$ exceeds that of case C180-3 by more than ten times. These two factors together lead to a decrease in the drag reduction efficiency of the DRL-based control strategy. Moreover, at higher Reynolds numbers, the control strategy optimized through DRL is less effective at impeding wall-normal momentum transport compared to traditional opposition control, as evidenced by the contrast of residual Reynolds stress. The main advantage of DRL optimization lies in its ability to effectively plan the wall blowing and suction in an expanded range, thereby elevating the virtual wall to a higher position and achieving better drag reduction efficiency.

Figure 7. Premultiplied spanwise energy spectra $k_{z}E_{uu}$ of streamwise velocity fluctuations $u^{\prime }$ under different control strategies. For $Re_{\tau }^{0}\thickapprox 180$ , (a) C180-0, (b) C180-opp, (c) C180-3. For $Re_{\tau }^{0}\thickapprox 550$ , (d) C550-0, (e) C550-opp, (f) C550-3. For $Re_{\tau }^{0}\thickapprox 1000$ , (g) C1000-0, (h) C1000-opp, (i) C1000-3.

To further quantify the impact of control strategies on the scales of the structures at different heights, especially the flow structures near the virtual wall, figure 7 presents the premultiplied energy spectra $k_{z}E_{uu}$ of $u^{\prime }$ . Here, $k_z$ is the spanwise wavenumber, and $\lambda _z=2\pi /k_z$ is the corresponding wavelength. In the near-wall region, the flow is dominated by streaks with spanwise scale $\lambda _{z}^{+}\approx 100$ and wall-normal height concentrated around $y^{+}=15$ , as depicted in figures 7(a), 7(d) and 7(g). After applying wall blowing and suction control, the peak velocity fluctuations in the near-wall region shift to a higher position, and a second spectral peak emerges in the viscous sublayer. These two peaks are separated by the virtual wall, as also suggested by Hammond et al. (Reference Hammond, Bewley and Moin1998). Notably, in cases utilizing the DRL-based control strategy (C180-3, C550-3 and C1000-3), the virtual wall is significantly higher than in cases using traditional opposition control, corroborating the conclusions drawn from table 4. The peak of velocity fluctuations corresponding to the near-wall streaks in the buffer layer also rises to a higher position. Furthermore, the intensity of velocity fluctuations in the viscous sublayer significantly increases after applying the DRL-based control strategy.

On the other hand, the characteristic scales of flow structures remain largely unchanged under different control strategies. The spanwise sizes of the near-wall streaks are consistently $\lambda _{z}^{+}\approx 100$ , while the peak of velocity fluctuations in the viscous sublayer due to wall blowing and suction stays within $\lambda _{z}^{+}=60$ – $80$ , i.e. slightly smaller than the spanwise sizes of the streaks. At high Reynolds numbers, wall blowing and suction have a trivial effect on the spanwise sizes of the outer large-scale structures, which remain at $\lambda _{z}\thickapprox O(h)$ . In the cases without control, the footprint of outer large-scale structures penetrates deeply into the near-wall region, as shown by the near-wall large-scale components in figures 7(d) and 7(g). This phenomenon, known as the superposition effect (Hoyas & Jiménez Reference Hoyas and Jiménez2006; Hutchins & Marusic, Reference Hutchins and Marusic2007b ; Mathis et al. Reference Mathis, Hutchins and Marusic2009; Marusic et al. Reference Marusic, Mathis and Hutchins2010a ), is noteworthy. After applying wall control, however, the footprint of outer large-scale structures cannot penetrate the virtual wall to reach the viscous sublayer or contribute to the residual velocity fluctuations on the virtual wall. This is particularly evident in cases C550-3 and C1000-3 using DRL models, as shown in figures 7(f) and 7(i). Thus the superposition effect does not directly cause the increasing residual Reynolds stress on the virtual wall at rising Reynolds numbers. Its impact on the decreasing drag reduction rate of the DRL models at high Reynolds numbers is also trivial.

Figure 8. Instantaneous distributions of $u^{\prime }$ on the $(x,z)$ plane at (a,c,e,g) $y^{+}=y_{vw}^{+}$ and (b,d,f,h) $y^{+}=150$ , for cases (a,b) C1000-opp, (c,d) C1000-1, (e,f) C1000-2, (g,h) C1000-3. The black rectangles represent some sample areas on the virtual wall where velocity fluctuations are stronger.

To identify the source of $- \langle u^{\prime }v^{\prime } \rangle _{vw}$ at high Reynolds numbers, figure 8 illustrates the distributions of streamwise velocity fluctuations at $y^{+}=y_{vw}^{+}$ and $y^{+}=150$ . The DRL-based control strategy reveals strong fluctuations on the virtual wall, characterized by clustered small-scale fluctuations concentrated in specific areas. Although these fluctuations are mitigated when the range of blowing and suction velocities is expanded, they remain stronger than those observed after opposition control. It shall be noted that these fluctuations are much smaller in size compared to the outer large-scale structures. Therefore, they are unlikely to be induced by the linear superposition effect, but are more plausibly related to the nonlinear amplitude modulation mechanism of the large-scale structures. As indicated by the black rectangles in figure 8, regions of strong fluctuations on the virtual wall often share similar spanwise locations with the outer large-scale high-speed regions, further supporting this point. In the streamwise direction, areas with clustered fluctuations are frequently situated upstream of the large-scale high-speed regions. This phenomenon can be attributed to the inclination angle of the large-scale coherent structures, as suggested by the near-wall fluctuation predictive models proposed by Marusic, Mathis & Hutchins (Reference Marusic, Mathis and Hutchins2010b ) and Mathis, Hutchins & Marusic (Reference Mathis, Hutchins and Marusic2011).

Further statistical evidence is required to support the relationship between the amplitude modulation of outer large-scale structures and the residual Reynolds stress at the virtual wall. The streamwise velocity fluctuations $u_{O}^{\prime }$ at the centre of the logarithmic region $y_{O}^{+}\approx 3.9\sqrt {Re_{\tau }}$ can be utilized to characterize outer large-scale structures (Mathis et al. Reference Mathis, Hutchins and Marusic2009, Reference Mathis, Hutchins and Marusic2011). A positive $u_{O}^{\prime }$ indicates a large-scale high-speed region, while a negative $u_{O}^{\prime }$ denotes a low-speed region. On the other hand, the residual fluctuations at the virtual wall exhibit a clustered distribution, as illustrated in figure 8. In areas with strong fluctuations, the streamwise and spanwise scales of the fluctuations are smaller, and the spatial alternation between positive and negative values is more pronounced. Considering the impact of spatial alternation, we select the envelope of the Reynolds stress at the virtual wall, denoted as $ |\mathcal {H} ( \langle u^{\prime }v^{\prime } \rangle _{vw} ) |$ , to measure the strength of the residual stress fluctuations, where $\mathcal {H}$ represents the operator of the two-dimensional Hilbert transform. Additionally, the inclination angle $\theta _{L}$ of the large-scale structures should be considered. The outer large-scale structures affecting the near-wall region are located downstream of this region. Hence we will examine primarily the relationship between the virtual wall fluctuations and $u^{\prime }_{O} (\Delta x_{m} )$ at a downstream displacement $\Delta x_{m}$ . Here, $\Delta x_{m}=(y_{O}-y_{vw})/\tan (\theta _{L})$ and $\theta _{L}=11^{\circ }$ – $15^{\circ }$ according to Mathis et al. (Reference Mathis, Hutchins and Marusic2011). We select $\theta _{L}=13^{\circ }$ for the subsequent discussions, noting that the results are robust within the range $\theta _{L}=11^{\circ }$ – $15^{\circ }$ .

Figure 9. Joint p.d.f. of the streamwise velocity fluctuations $u^{\prime }_{O}(\Delta x_{m})$ at the centre of the logarithmic region and the envelope of Reynolds stress $|\mathcal {H}(\langle u^{\prime }v^{\prime }\rangle _{vw})|$ at the virtual wall, for cases (a) C1000-1, (b) C1000-2, (c) C1000-3. Contour levels are $0.1 (0.1) 0.8$ of the maximum probability density.

The joint p.d.f. between the outer $u^{\prime }_{O} (\Delta x_{m} )$ and the envelope of Reynolds stress at the virtual wall is depicted in figure 9. At the position where $ |\mathcal {H} (\langle u^{\prime }v^{\prime } \rangle _{vw} ) |$ approaches $0$ , the joint p.d.f. tilts to the left, indicating negative $u^{\prime }_{O}$ in low-speed large-scale motions. As the envelope of Reynolds stress gradually increases, the joint p.d.f. shifts, tilting to the right, which is particularly evident in the upper half of the distribution. This pattern suggests that locations with strong residual Reynolds stress fluctuations are typically situated below large-scale high-speed regions, while areas with weaker residual Reynolds stress generally correspond to large-scale low-speed regions. This observation is consistent with the findings illustrated in figure 8. As the range of blowing and suction velocities is extended, although the intensity of $- \langle u^{\prime }v^{\prime } \rangle _{vw}$ diminishes, the influence of outer large-scale structures on the distribution of Reynolds stress remains nearly unchanged. This further substantiates the relationship between the amplitude modulation of outer large-scale structures and the residual Reynolds stress at the virtual wall.

In summary, compared to the traditional opposition control method, the DRL-based control strategy demonstrates superior drag reduction capabilities by effectively elevating the virtual wall to a higher position. As the range of blowing and suction velocities is expanded, the virtual wall ascends further and the residual Reynolds stress on the virtual wall decreases, both of which enhance the drag reduction rate of the DRL models. However, as the Reynolds number increases, large-scale structures emerge in the outer region. Their amplitude modulation effect significantly increases the residual Reynolds stress on the virtual wall, and disrupts the virtual wall’s blockage in large-scale high-speed regions, thereby reducing the drag reduction rate of the DRL models.

3.4. Dynamic analysis of drag reduction using budget equations

In the previous subsection, the drag reduction mechanism was examined from a kinematic perspective using the virtual wall theory. This subsection will further discuss the dynamics mechanism behind drag reduction based on the analysis of budget equations.

According to the FIK identity proposed by Fukagata et al. (Reference Fukagata, Iwamoto and Kasagi2002), the skin frictions in the current cases are primarily attributed to the Reynolds shear stress $ \langle -u^{\prime }v^{\prime } \rangle$ . Hence it is necessary to discuss how the DRL-based control strategies reduce the drag by altering $ \langle -u^{\prime }v^{\prime } \rangle$ . The transport equation of the Reynolds stress $ \langle -u^{\prime }v^{\prime } \rangle$ is written as

(3.3) \begin{equation} \begin{aligned} \frac {\partial }{\partial t}\left \langle -u^{\prime }v^{\prime }\right \rangle =& \underbrace {\left \langle v^{\prime }v^{\prime }\right \rangle \frac {{\rm d}U}{{\rm d}y}}_{P_{12}} +\underbrace {\frac {{\rm d}}{{\rm d}y}\left \langle u^{\prime }v^{\prime }v^{\prime }\right \rangle }_{D_{12,t}} +\underbrace {\left (-\mu\, \frac {{\rm d}^{2}\left \langle u^{\prime }v^{\prime }\right \rangle }{{\rm d}y^{2}}\right )}_{D_{12,\nu }}\\ & {}+\underbrace {\frac {1}{\rho }\left \langle v^{\prime }\,\frac {\partial p^{\prime }}{\partial y}+u^{\prime }\,\frac {\partial p^{\prime }}{\partial x}\right \rangle }_{VP_{12}} +\underbrace {2\mu \left \langle \frac {\partial u^{\prime }}{\partial x_{j}}\,\frac {\partial v^{\prime }}{\partial x_{j}}\right \rangle }_{\varepsilon _{12}}, \end{aligned} \end{equation}

where $P_{12}$ is the turbulent production, $D_{12,t}$ is the turbulent diffusion, $D_{12,\nu }$ is the viscous diffusion, $VP_{12}$ is the velocity pressure-gradient term, and $\varepsilon _{12}$ is the dissipation. Here, $U$ is the mean streamwise velocity.

Figure 10. Wall-normal distributions of the budget terms of Reynolds shear stress $\langle -u^{\prime }v^{\prime }\rangle$ in (3.3): $P_{12}$ , $VP_{12}$ , $\varepsilon _{12}$ , $D_{12,t}$ , $D_{12,\nu }$ , for (a,b) C180, (c,d) C550, (e,f) C1000. Lines without markers indicate cases with suffix 0; plus signs indicate cases with suffix 1; circles indicate cases with suffix 2; triangles indicate cases with suffix 3.

Figure 10 shows the wall-normal distributions of the budget terms on the right-hand side of (3.3). In the budget terms of the Reynolds shear stress, the production $P_{12}$ and the velocity pressure-gradient term $VP_{12}$ are significantly stronger and more dominant compared to the other terms, as suggested in figure 10(a,c,e). The viscous diffusion $D_{12,\nu }$ , although large and negative in the viscous sublayer, decays rapidly above $y^+=5$ to become smaller than the dominant terms. Furthermore, the dissipation $\varepsilon _{12}$ and the turbulent diffusion $D_{12,t}$ are much smaller than the other terms, and their contribution could be considered negligible. Among the two dominant terms, the velocity pressure-gradient term $VP_{12}$ , which can be further divided into the pressure diffusion and the redistribution, mainly represents the transport of Reynolds stress at different heights, and the redistribution among different components caused by pressure. And it primarily acts as a negative term to offset the production $P_{12}$ , which remains positive and determines the magnitude of the Reynolds shear stress.

In the uncontrolled cases, the turbulent production $P_{12}$ increases with height, reaching a peak at approximately $y^{+}=15$ – $20$ , and then decreases continuously. After implementing wall blowing and suction, $P_{12}$ at the wall is no longer zero, leading to a significant increase in $P_{12}$ within the viscous sublayer. This results in a larger Reynolds stress in the viscous sublayer compared to that in the uncontrolled cases, as depicted in figure 5. However, this effect is confined to the narrow height range of the viscous sublayer, and has a limited impact on the overall skin friction. In the cases with control, $P_{12}$ decreases rapidly with height, and reaches a trough at approximately $y^{+}=10$ . In the range $10\lt y^{+}\lt 20$ , $P_{12}$ is significantly smaller than in the uncontrolled case, corresponding to a lower Reynolds shear stress in figure 5. As the range of blowing and suction velocities is expanded, the height corresponding to the trough gradually increases, and $P_{12}$ at the trough further decreases, leading to a reduction in Reynolds shear stress. Moreover, the decrease in $P_{12}$ near the trough compared with the uncontrolled case is less pronounced at higher Reynolds numbers, as indicated in figure 10(e). This results in a reduced suppression effect on $ \langle -u^{\prime }v^{\prime } \rangle$ at higher Reynolds numbers, shown in figure 5(c), further leading to a decreased drag reduction rate. As the height increases, $P_{12}$ in the controlled case gradually rises above $y^{+}=15$ , peaks, and then decreases continuously, eventually collapsing with the uncontrolled case in the outer region.

According to (3.3), the turbulent production $P_{12}$ of Reynolds shear stress consists of two parts: $ \langle v^{\prime }v^{\prime } \rangle$ and ${\rm d}U/{\rm d}y$ . The latter could be viewed as the outcome associated with changes in Reynolds shear stress and skin friction. Therefore, the following discussion will focus primarily on the wall-normal kinetic energy $ \langle v^{\prime }v^{\prime } \rangle$ to identify the source of changes in $P_{12}$ . The wall-normal distributions of $v_{rms}$ under different control strategies have already been shown and discussed in figure 4. After adopting the DRL models, the wall-normal velocity fluctuations in the buffer layer gradually decrease. As the range of blowing and suction velocities is extended, $v_{rms}$ continues to decrease, but this decreasing trend slows down with increasing Reynolds number. This is similar to the evolution trend of the turbulent production $P_{12}$ . The transport equation of the wall-normal kinetic energy $ \langle v^{\prime }v^{\prime } \rangle$ is written as

(3.4) \begin{equation} \begin{aligned} \frac {\partial }{\partial t}\left \langle v^{\prime }v^{\prime }\right \rangle =& \underbrace {-\frac {{\rm d}}{{\rm d}y}\left \langle v^{\prime }v^{\prime }v^{\prime }\right \rangle }_{D_{22,t}} +\underbrace {\mu\, \frac {{\rm d}^{2}\left \langle v^{\prime }v^{\prime }\right \rangle }{{\rm d}y^{2}}}_{D_{22,\nu }} +\underbrace {\left (-\frac {2}{\rho }\,\frac {{\rm d}}{{\rm d}y}\left \langle p^{\prime }v^{\prime }\right \rangle \right )}_{D_{22,p}}\\ & {}+\underbrace {\frac {2}{\rho }\left \langle p^{\prime }\,\frac {\partial v^{\prime }}{\partial y}\right \rangle }_{\Phi _{22}} +\underbrace {\left (-2\mu \left \langle \frac {\partial v^{\prime }}{\partial x_{j}}\,\frac {\partial v^{\prime }}{\partial x_{j}}\right \rangle \right )}_{\varepsilon _{22}}, \end{aligned} \end{equation}

where $D_{22,t}$ is the turbulent diffusion, $D_{22,\nu }$ is the viscous diffusion, $D_{22,p}$ is the pressure diffusion, $\Phi _{22}$ is the redistribution, and $\varepsilon _{22}$ is the dissipation.

Figure 11. Wall-normal distributions of the budget terms of wall-normal kinetic energy $\langle v^{\prime }v^{\prime }\rangle$ in (3.4): $\Phi _{22}$ , $\varepsilon _{22}$ , $D_{22,p}$ , $D_{22,t}$ , $D_{22,\nu }$ , for (a,b) C180, (c,d) C550, (e,f) C1000. Lines without markers indicate cases with suffix 0; plus signs indicate cases with suffix 1; circles indicate cases with suffix 2; triangles indicate cases with suffix 3.

The wall-normal distributions of the budget terms on the right-hand side of (3.4) are illustrated in figure 11. Among these budget terms, the wall-normal kinetic energy $ \langle v^{\prime }v^{\prime } \rangle$ is influenced predominantly by redistribution $\Phi _{22}$ , pressure diffusion $D_{22,p}$ , and dissipation $\varepsilon _{22}$ . Conversely, the effects of turbulent diffusion $D_{22,t}$ and viscous diffusion $D_{22,\nu }$ are comparatively minor. In the viscous sublayer, the redistribution $\Phi _{22}$ is primarily negative, indicating that the wall-normal velocity fluctuations are being redistributed to other directions. This negative contribution is offset mainly by the positive pressure diffusion $D_{22,p}$ . As the height increases, the redistribution $\Phi _{22}$ changes from negative to positive, indicating that the wall-normal velocity fluctuations are absorbing TKE from other components. Meanwhile, the pressure diffusion $D_{22,p}$ rapidly decreases and gradually approaches zero above $y^{+}=20$ . On the other hand, the dissipation $\varepsilon _{22}$ gradually increases, acting as a negative term to offset the positive contribution from the redistribution $\Phi _{22}$ . Based on the previous discussion, the drag reduction achieved by the DRL model stems mainly from the dynamic changes in the buffer layer, while significant changes in the viscous sublayer contribute very little to the overall drag reduction. Among the three dominant terms, the pressure diffusion $D_{22,p}$ primarily represents the TKE transport caused by pressure at different heights, with its intensity decreasing significantly in the buffer layer compared to the viscous sublayer. Therefore, in the following discussion, we will focus primarily on the redistribution $\Phi _{22}$ , which represents the exchange mechanism between wall-normal velocity fluctuations and other velocity components.

Compared to the uncontrolled cases, the DRL-based control strategy causes a significant decrease in $\Phi _{22}$ in the buffer layer, and raises the position where it changes from negative to positive to approximately $y^+=20$ . This leads to less kinetic energy being transferred to the wall-normal velocity fluctuations in the buffer layer, thereby suppressing the production of Reynolds stress. The suppressing effect of the DRL models on the redistribution $\Phi _{22}$ increases further as the range of blowing and suction velocities is extended. At a low Reynolds number $Re_{\tau }^{0}\thickapprox 180$ , this decreasing trend can extend to the logarithmic layer, as illustrated in figure 11(e). However, as the Reynolds number increases, the reduction in the redistribution $\Phi _{22}$ above $y^+=30$ nearly vanishes, and the suppression of $\Phi _{22}$ in the buffer layer by the DRL models becomes weaker. This change corresponds to the decrease in drag reduction rate at higher Reynolds numbers.

It is important to note that due to the incompressibility condition (where divergence equals zero), the sum of the redistribution terms for the three velocity components, namely $\Phi _{11}=(2/\rho )\langle p^{\prime }\,\partial u^{\prime }/\partial x\rangle$ , $\Phi _{22}=(2/\rho )\langle p^{\prime }\,\partial v^{\prime }/\partial y\rangle$ and $\Phi _{33}=(2/\rho )\langle p^{\prime }\,\partial w^{\prime }/\partial z\rangle$ , is $0$ . Above the viscous sublayer, the TKE is typically redistributed from the streamwise component to the wall-normal and spanwise components (Lee & Moser Reference Lee and Moser2019), also indicated by the positive $\Phi _{22}$ observed in figure 11. This redistribution corresponds to the transient growth of the streamwise velocity streaks associated with $u^{\prime }$ , leading to the generation of quasi-streamwise vortices associated with $v^{\prime }$ and $w^{\prime }$ in the near-wall turbulent self-sustaining cycle. Thus the observed weakening of $\Phi _{22}$ due to the DRL-based control strategy can be interpreted as the suppression of the near-wall self-sustaining mechanism. Consequently, a larger proportion of TKE would remain in the streamwise component, resulting in smoother streak structures. Figure 12 illustrates the instantaneous distributions of $u^{\prime }$ in the near-wall region. The controlled cases exhibit significantly smoother near-wall streaks compared to the uncontrolled results, and this trend is consistent across different Reynolds numbers. This observation aligns with the suppressed near-wall self-sustaining mechanism, and also reflects the DRL model’s influence on $\Phi _{22}$ from the perspective of flow structures.

Figure 12. Instantaneous distributions of $u^{\prime }$ on the $(x,z)$ plane at $y^{+}=20$ , for cases (a) C180-0, (b) C180-3, (c) C1000-0, (d) C1000-3.

Moreover, the streamwise velocity fluctuations merit further discussion due to their significant impact from the DRL-based control strategy, as illustrated in figure 4. The transport equation of the streamwise kinetic energy $ \langle u^{\prime }u^{\prime } \rangle$ is expressed as

(3.5) \begin{equation} \begin{aligned} \frac {\partial }{\partial t}\left \langle u^{\prime }u^{\prime }\right \rangle =& \underbrace {-2\left \langle u^{\prime }v^{\prime }\right \rangle \frac {{\rm d}U}{{\rm d}y}}_{P_{11}} +\underbrace {\left (-\frac {{\rm d}}{{\rm d}y}\left \langle u^{\prime }u^{\prime }v^{\prime }\right \rangle \right )}_{D_{11,t}} +\underbrace {\mu\, \frac {{\rm d}^{2}\left \langle u^{\prime }u^{\prime }\right \rangle }{{\rm d}y^{2}}}_{D_{11,\nu }}\\ & {}+\underbrace {\frac {2}{\rho }\left \langle p^{\prime }\,\frac {\partial u^{\prime }}{\partial x}\right \rangle }_{\Phi _{11}} +\underbrace {\left (-2\mu \left \langle \frac {\partial u^{\prime }}{\partial x_{j}}\,\frac {\partial u^{\prime }}{\partial x_{j}}\right \rangle \right )}_{\varepsilon _{11}}, \end{aligned} \end{equation}

where $P_{11}$ is the turbulent production, $D_{11,t}$ is the turbulent diffusion, $D_{11,\nu }$ is the viscous diffusion, $\Phi _{11}$ is the redistribution, and $\varepsilon _{11}$ is the dissipation.

Figure 13. Wall-normal distributions of the budget terms of streamwise kinetic energy $\langle u^{\prime }u^{\prime }\rangle$ in (3.5): $P_{11}$ , $\Phi _{11}$ , $\varepsilon _{11}$ , $D_{11,t}$ , $D_{11,\nu }$ , for (a,b) C180, (c,d) C550, (e,f) C1000. Lines without markers indicate cases with suffix 0; plus signs indicate cases with suffix 1; circles indicate cases with suffix 2; triangles indicate cases with suffix 3.

Figure 13 illustrates the wall-normal distributions of the budget terms on the right-hand side of (3.5). Among these budget terms, the production $P_{11}$ plays a crucial role, consistently remaining positive across various heights. In contrast, the terms $D_{11,t}$ and $D_{11,\nu }$ are relatively smaller, and represent primarily the transport of TKE in the wall-normal direction. The dissipation term $\varepsilon _{11}$ remains negative at different heights, counterbalancing the production $P_{11}$ . Additionally, unlike $\Phi _{22}$ associated with wall-normal velocity fluctuations, the redistribution term $\Phi _{11}$ has a trivial impact on the streamwise TKE. In the uncontrolled cases, the production term $P_{11}$ gradually increases with height, reaching a peak at approximately $y^{+}=10$ – $15$ . This height is similar to the $u_{rms}$ peak in the near-wall region, as depicted in figure 4. After applying control, this peak disappears and transforms into a trough. The production term $P_{11}$ in the viscous sublayer increases significantly compared with the uncontrolled case. In the buffer layer, $P_{11}$ initially decreases and reaches the trough, then increases with height, with a second peak appearing at approximately $y^{+}=20$ . It can be observed that after applying the DRL model, the trend of $P_{11}$ changes in a manner highly consistent with the changes in $u_{rms}$ . Notably, the turbulent production $P_{11}$ of the streamwise TKE consists mainly of two parts: $ \langle -u^{\prime }v^{\prime } \rangle$ and ${{\rm d}U}/{{\rm d}y}$ . The latter can be considered as a result of changes in Reynolds stress. Therefore, we can infer that the significant changes in $u_{rms}$ caused by the DRL-based control are primarily due to alterations in Reynolds shear stress.

In summary, figure 14 illustrates the dynamic mechanism through which DRL-based control strategies influence skin friction. The application of wall blowing and suction, directed by the DRL models, effectively suppresses the near-wall self-sustaining process, thereby leading to smoother velocity streaks. This suppression manifests as a decrease in the redistribution term of wall-normal TKE within the buffer layer, consequently reducing wall-normal velocity fluctuations. The reduction in $ \langle v^{\prime }v^{\prime } \rangle$ further diminishes the production term of Reynolds stress, resulting in a decrease in $ \langle -u^{\prime }v^{\prime } \rangle$ . Ultimately, this decline in Reynolds stress results in a reduction of skin friction. Moreover, the weakening of streamwise velocity fluctuations can also be attributed to the decrease in Reynolds stress. When the range of blowing and suction velocities is expanded, the aforementioned effects are amplified, leading to an increase in the extent of drag reduction. Conversely, as the Reynolds number rises and drag reduction diminishes, these trends are reversed.

Figure 14. Schematic diagram of the dynamic mechanism through which DRL-based control strategies influence skin friction.