Hamilton-jacobi Equation for Optimal Control of Nonlinear Stochastic Distributed Parameter Systems Applied to Air Pollution Process

This paper derives Hamilton-Jacobi equation (HJE) in Hilbert space for optimal control of stochastic distributed parameter systems (SDPSs) governed by partial differential equations (SPDEs) subject to both state-dependent and additive stochastic disturbances. First, nonlinear SDPSs are transformed to stochastic evolution systems (SESs), which are governed by stochastic ordinary differential equations (SODEs) in Hilbert space, using functional analysis. Second, the Hamilton-Jacobi equation (HJE), of which the solution results in an optimal control law, is derived. Third, a problem of optimal control of linear SDPSs, which include the air pollution process, with a quadratic cost functional is addressed as an application of the HJE. After, the control design is done, the SESs are transformed back to Euclidean space for implementation.

The first approach, referred to as the modal control one, discretizes the PDEs to obtain lumped-parameter systems described in terms of modal coordinates, i.e., systems of ordinary differential equations (ODEs), to which the classical control design methods [1], [21], [22] can be applied.The modal control approach, see [30], [35], [18], can only control a certain number of modes of a DPS, and has difficulty in computing appropriate gain matrices.
The second approach applies the calculus of variations [8], [19] to derive a set of Euler-Lagrange (EL) equations in a form of the two-point boundary value (TPBV) problem, of which the solution results in optimal control inputs.Several techniques proposed to solve the TPBV problem include [26], [31] on hill-climbing algorithms, and [20], [32], [17], [3], [4] on an extension of the representer method to nonlinear systems based on linearization.However, the hill-climbing algorithms require a search in the whole control trajectories and the extended representer method requires many representers.To overcome the above issue, a non-climbing algorithm was proposed in [13] to solve a general TPBV problem.
On the other hand, the use of tools from functional analysis makes it possible to represent PDEs in Euclidean space as evolution systems, i.e., systems governed by ODEs in Hilbert space.Thus, we can utilize various ideas from control design methods well developed for systems governed by ODEs in Euclidean space.After nonlinear SDPSs are transformed to SESs in Hilbert space by using functional analysis, the goal of this paper is to present a derivation of the HJE in Hilbert space, which is relatively easy to follow, for optimal control of SDPSs subject to both state-dependent and additive stochastic disturbances.Several remarks on applications of the derived HJE are then made in relation to optimal control design procedure for SESs, stability analysis of SESs, and Itô's formula frequently used in control of stochastic systems.Finally, the HJE is applied to solve the optimal control problem of linear SDPSs, which include the air pollution process, with a quadratic cost functional.

Problem formulation
Let D be a open bounded set in Euclidean r-space R r with piecewise smooth boundary S, and let t denote time defined on an interval T = [t 0 , t f ] with t f > t 0 ≥ 0. In this paper, we consider the following SDPS in the Euclidean space: defined for t ∈ T , where x = col(x 1 , ..., x r ) ∈ D is the r-dimensional spatial coordinate vector; X(x, t) is the n-dimensional vector function describing the system state; w(x, t) is p-dimensional stochastic disturbance; the dot over w(x, t) denotes the formal derivative with respect to time t; N (•) and N b (•) are the n-dimensional vector functions; G(•) are the n × p matrix function; X 0 (•) is the n-dimensional Gaussian random vector with zero mean and the covariance matrix P 0 (x, y); n is the outward directed normal vector to the boundary S; In (1), the following notations have been used where r i ≥ 0, and k .are nonnegative integers; and cos(n, x i ) is the direction cosine of x i .We assume that the p-dimensional stochastic disturbance vector w(•) is a Wiener process with zero mean value and the covariance matrix function given by Hence, ẇ(x, t) can be regarded as the white Gaussian noise in t [9].The control objective is to design the control inputs u d (•) and u b (•)) so as to minimize the cost functional: where E (X,t {•} is the conditional expectation of {•} with respect to the σ-field generated by {(X(x, τ ), t 0 ≤ τ ≤ t}; and J f (•) and L(•) are the positive and real-valued integrable functions with respect to time t.

Transformation of SDPS in Euclidean to SES in Hilbert space
In this section, functional analysis [41] is used to transform the SDPS (1) in the Euclidean space to a SES in the Hilbert space.As such, we first define L 2 n (D) and H m n (D) spaces by where ∥ • ∥ n is the n-dimensional Euclidean norm; |α| = k 1 + ... + k r ≤ m for α = (k 1 , ..., k r ) with k 1 , ..., k r nonnegative integers.The spaces L 2 n (S) and H m n (S) are defined similarly by replacing D by S in (6).We now define the norms of L 2 n (D) and H m n (D) by Thus, L and ∥ϕ(x, t)∥ H m n (D) , respectively, given in (7).Considering x ∈ D as the parameter defined at very t ∈ T , we can regard X(x, t) as , respectively.Therefore, the SDPS (1) can be described as which is referred to as the SES in the Hilbert space.In (8), showing the aggregate of bounded linear mapping from U 2 into U 6 .
The covariance matrix function of w(x, t) is described from ( 3) by where Q is a compact, positive, bounded, and trace class operator mapping

and h are any elements of the Hilbert space H , then h
where

Hamilton-Jacobi equation in Hilbert space
In this section, we derive the HJE for the SES (8).As such, Let W (X(t), t) be the minimum of Let us assume that W (X(t), t) is continuously differentiable on T and continuously twice Fréchet differentiable on U 1 .Applying Bellman's optimality principle [2] to the integral term of ( 11) results in min where with σ > 0. Applying the mean-value theorem to the term L 1 yields where 0 < ε < 1.On the other hand, applying the Taylor expansion theorem [11] to the term L 2 results in where W X (•) and W XX (•) are the first and the second order Fréchet derivatives on U 1 , which are a linear mapping from U 1 into the real-valued R, and a bilinear mapping from U 1 × U 1 into R, respectively, and O(•) is the same order infinitesimal, and ∆X(t) = X(t + σ) − X(t).(17) We now calculate the term L 2 in (16).From (8), we have where Φ is a a bilinear mapping from U 1 × U 1 into R and {λ i , e i , i = 1, 2, ...} is an orthonormal set of eigenvalues and eigenfunctions of the operator Q, i.e., Substituting ( 18) into ( 16) yields On the other hand, from ( 8), ( 13), (14), and (15) we have Substituting ( 15) and (20) with O(∥∆X(t)∥ 3 U 6 ) satisfied ( 21) into (13) yields From the Bochner theorem [41], we have Using this identity and letting σ → 0 in (22) result in the following HJE in the Hilbert space: ] , (24) where H(•) is referred to as the system Hamiltonian, which is a real-valued functional given by with Tr[•] and the asterisk being the trace and the adjoint of the corresponding operator, respectively.

Remark 4.1
If the functional space is replaced by a space of finite dimension, then N (•) and G(•) become a vector and a matrix in the finite dimensional space, respectively, and the HJE ( 24) is reduced to the HJE derived for stochastic lumpedparameter systems (SLPSs) in Euclidean space [24].Moreover, if the SDPS ( 8) is linear subject to additive stochastic noise, the HJE ( 24) is reduced to the results in [39].In general, the first two steps of the above procedure are formidable for a predefined cost functional J(•) in (11) and a general nonlinear SES in (8).This formidable task was tackled by approximating methods on either dealing with the above procedure directly or converting the HJE (24) to a two-point boundary value problem, see [26,31] on hill-climbing algorithms; [3,17,20,32] on an extension of the representer method [3] to nonlinear systems based on linearization; and [13] on a non-clinbing algorithm.Moreover, no formal proof of convergence of the approximating solutions by the above approximating methods to the true ones even though there only are several works on existence and uniqueness of (weak/viscosity) solutions of HJEs to certain SLPSs and SDPSs [7,29,38].

Remark 4.3
If the controls u d (t) and u b (t) in the SES ( 8) are set to zero or are designed such that they are functions of X(t) and t, the HJE ( 24) can be used for stability analysis of the SES (8) without controls or of the resulting closed loop system.This was used in [12] for stability analysis of linear DPSs.Moreover, if the SES (8) does not contain the controls u d (t) and u b (t), and the cost functional J(•) is chosen as the HJE (24) reduces to Itô's formula [34].
5 Application to optimal control of an air pollution process

Mathematical model of an air pollution process
Let D be a open bounded set in Euclidean three-dimensional space R 3 with piecewise smooth boundary S, let t denote time defined on an interval T = [t 0 , t f ] with t f > t 0 ≥ 0, and let r = col(x, y, z).An air pollution process can be modeled by a system of advection-diffusion equations, which are a set of the following partial differential equations in three dimensional space as follows [37]: where c(r, t) = col(c 1 (r, t), ..., c n (r, t)) is the n-dimensional vector of pollutant concentrations in location r and at time t measured in parts per million (p.p.m); V x , V y and V z are wind velocities in the x, y and z directions, respectively (m/s); K x , K y and K z are turbulent diffusivities which are simplycalled diffusion coefficient matrices in the x, y and z directions, respectively (m 2 /s); Kx , Ky and Kz are the mean parts of K x , K y and K z , respectively; K is the deposition coefficient matrix; Kb is a constant parameter matrix; f (r, t) represents the emission source; l x , l y and l z are direction cosines of the outward normal to the boundary surface S; Now the wind velocities (V x , V y , V z ), and the turbulent diffusivities (K x , K y , K z ) are decomposed of their mean and stochastic parts as follows where • and W • denote the mean and stochastic parts of •, respectively.Substituting (28) into (27) It is now seen that the air pollution process has been rewritten as a system of stochastic PDEs.In the next subsection, we will present an application of the control design in the previous sections to a fairly general linear SDPS, which covers the stochastic air pollution process (29).The control objective is to design the controls u d and u b so as to minimize the pollutant concentration c(r, t) while putting appropriate weights on the controls u d and u b .

Linear stochastic distributed parameter system
Consider the following linear SDPS subject to both state-dependent and additive stochastic disturbances: where where the , and A j (ξ, t) is given by with n being the outward normal to the boundary S at the point ξ ∈ S, and (n, x i ) being the angle between the outward normal n and the x i -axis.In (30), G 0 (x, t) is an n-dimensional vector function; G 1 (x, t) is a n × n matrix function; B d (x, t) and B b (ξ, t) are n × m d and n × m b matrix functions; w(x, t) = col(w 1 (x, t), w 0 (x, t)) is mutually independent Wiener process, of which intensity Q(x, y) of the covariance matrix function defined by ( 3) is given by Moreover, the functionals J f (•) and L(•) in ( 5) are specified in a quadratic form as where

Transformation to SES in Hilbert space
Applying the procedure in the previous section, we can transform the linear SDPS (30) to the following SES in Hilbert space: and the functionals J f (•) and L(•) from ( 34) are transformed to Therefore, the Hamiltonian H(•) of the system ( 35) is given from ( 25) by ] . (37)

Control design
To design the controls u d (t) and u b (t), we need to solve the HJE (24) with H(•) given in (37).Let us assume that W (X(t), t) is of the following form where ) is a self-adjoint positive definite trace class operator with the kernel P (x, y, t) and P 0 (t) is a positive and real-valued function of t.From (38), the first and second Fréchet derivatives W X (X, t) and W XX (X, t) are given by W X (X(t), t) = P (t)X(t), W XX (X(t), t) = P (t).
Substituting ( 39) into (37) results in ] . ( Applying Green's formula, see Lemma 3.1 in [14] to ⟨ P (t)X(t), A(t)X(t) ⟩ where A * (t) and β * (t) are given by Substituting (40) with the use of ( 41) into the HJE (24) yields the following optimal controls and P (t) and P 0 (t) must satisfy the following equations: where (A * (t)P (t)) * and P b (t) are the adjoint of A * (t)P (t) and an element of L (L 2 n (S), L 2 n (S)), respectively, and satisfy: (45)

Transformation of SES in Hilbert space to PDE in Euclidean space
For implementation, we now need to transform the controls u d (t) and u b (t) in (43) and P (t) and P 0 (t) in (44) to Euclidean space.For symmetric operators, the following two equations hold from the Schwartz kernel theorem [28]: where ϕ(t) is an infinitely differentiable function with compact support in D, P (x, y, t) and P (x, y, t) are kernels of P (t) and P −1 (t), respectively.With (46), the controls u d (t) and u b (t) in (43) are transformed to Euclidean space as follows: Moreover, P (t) and P 0 (t) in (44) are transformed to Euclidean space as follows: (48)

Finite number of controls
We assume that there are M d controllers at fixed points x 1 , .
where δ(•) is the Dirac delta function, we can use the same control design procedure in the previous subsections to obtain the following optimal controls:

Conclusions
This paper has presented a procedure to transform the nonlinear SDPSs in the Euclidean space to the SESs in the Hilbert space.Then, the HJE for the SESs was derived for solving the optimal control problem of nonlinear SDPSs under both state-dependent and additive stochastic disturbances.The HJE was applied to solve the optimal control problem of linear SDPSs, which include the stochastic air pollution process.The designed control laws were then transformed back to Euclidean space for implementation using the Schwartz kernel theorem.The proposed optimal control design can be applied to further improve performance of controlling other practical DPSs such as marine riser systems in [15], [16], and [33].

Remark 4 . 2
The optimal controls u d (t) and u b (t) can be found by performing the following three-step procedure:1.Minimize the Hamiltonian H(•) to obtain u d (•) and u b (•) as functions of X(t),t, andW (t), i.e., u d (•) = ϖ d (X(t), t, W (t)) and u b (•) = ϖ b (X(t), t, W (t));2. Substitute the above ϖ d (•) and ϖ b (•) into(24), then solve for W (•), which must satisfy the aforementioned conditions such as the boundary defined in the third equation of (8) and twice Fréchet differentiable; 3. Substitute the found W (•) into the expressions of ϖ d (•) and ϖ b (•) to obtain the controls u d (•) and u b (•).
B d (r, t) and B b (r, t) are n × m d and n × m b matrix functions; u d and u b are m d -dimensional and m b -dimensional control input vectors.It is noted that the controls u d and u b can be either distributed or discrete over the domain D and the boundary S, see Subsection 5.6.
y) and Ω(x, y) are n × n positive definite matrix functions; and R d (x, y) and R b (ξ, η) are m d × m d and m b × m b positive definite matrix functions, respectively.It is clearly seen that the linear SDPS (30) includes the stochastic air pollution process (29).