Looks like there are a a few plus signs missing. I assume the integral is supposed to read
[tex]\displaystyle \iint_R (4x^2 - 5xy + 4y^2) \, dA[/tex]
and the boundary of R is the ellipse 4x² - 5xy + 4y² = 2.
When we make the given substitution,
x = 2/3 u - 2/13 v
y = 2/u + 2/13 v
the ellipse equation transforms to
4 (2/3 u - 2/13 v)² - 5 (2/3 u - 2/13 v) (2/3 u + 2/13 v) + 4 (2/3 u + 2/13 v)² = 2
4 (4/9 u² - 8/39 uv + 4/169 v²) - 5 (4/9 u² - 4/169 v²) + 4 (4/9 u² + 8/39 uv + 4/169 v²) = 2
4/3 u² + 4/13 v² = 2
2/3 u² + 2/13 v² = 1
and this is yet another ellipse that forms the boundary of a new region R'.
Under this change of coordinates, the Jacobian is
[tex]J = \begin{bmatrix}x_u & y_u \\ x_v & y_v\end{bmatrix} = \begin{bmatrix}\frac23 & \frac23 \\ -\frac2{13} & \frac2{13}\end{bmatrix}[/tex]
and det(J) = 8/39. Then the integral transforms to
[tex]\displaystyle \frac8{39} \iint_{R'} \left(\frac43 u^2 + \frac4{13} v^2\right) \, dA[/tex]
Compute the integral in polar coordinates, setting
u = √3/2 r cos(θ)
v = √13/2 r sin(θ)
This introduces yet another Jacobian,
[tex]J = \begin{bmatrix}u_r & v_r \\ u_\theta & v_\theta\end{bmatrix} = \begin{bmatrix}\frac{\sqrt3}2\cos(\theta) & \frac{\sqrt{13}}2\sin(\theta) \\ -\frac{\sqrt3}2 r\sin(\theta) & \frac{\sqrt{13}}2 r\cos(\theta)\end{bmatrix}[/tex]
so that det(J) = √39/4 r, while the integrand changes to
4/3 (√3/2 r cos(θ))² + 4/13 (√13/2 r sin(θ))²
4/3 (3/4 r² cos²(θ)) + 4/13 (13/4 r² sin²(θ))
r² cos²(θ) + r² sin²(θ)
r²
Then the integral becomes
[tex]\displaystyle \frac8{39} \cdot \frac{\sqrt{39}}4 \iint_{R'} r^2 \cdot r \, dA = \frac2{\sqrt{39}} \int_0^{2\pi} \int_0^1 r^3 \, dr \, d\theta[/tex]
Computing the integral is trivial.
[tex]\displaystyle \frac2{\sqrt{39}} \int_0^{2\pi} \int_0^1 r^3 \, dr \, d\theta = \frac2{\sqrt{39}} \underbrace{\left(\int_0^{2\pi} d\theta\right)}_{2\pi} \underbrace{\left(\int_0^1 r^3 \, dr\right)}_{1/4}[/tex]
[tex]\displaystyle = \boxed{\frac{\pi}{\sqrt{39}}}[/tex]
