Integration (as also differentiation) is one of the essential tools in a machine learning practitioner's toolbox. One of the primary techniques utilized in integration is integration by parts, and to high schoolers applications of integration by parts often appear somewhat magical. The material for this article arose while discussing the problem with a friend in high school. The desire in this article is to

- Show how to utilize integration by parts, motivate why the need.
- Show how to think about this problem in multiple ways.

- We will look at our repertoire of techniques and try to form-fit some of these for the problem.
- This will not entirely work out, so we will start exploring tweaks in order to solve the problem.
- The
**icing**: after solving the original problem, we will not rest satisfied. This will lead us to other interesting solutions, wherein the techniques can be used in many other places.

We will assume here that the reader is aware of some of the more effective techniques in integration: integration by parts and integration by substitution.

Out of the two techniques mentioned, **integration by substitution** is perhaps the
simpler one. It is a kind of form-fitting where it leaps to the eye that a
smart substitution can solve the integral for us. For instance, looking at
$$
\int \frac{\ln x}{x}\, dx
$$
we might realize that $\frac{d}{dx} (\ln x) = \frac{1}{x}$, and so substituting
$t = \ln x$ will solve the problem for us (I leave it to the reader to complete
solving this indefinite integral).

**Integration by parts**, in comparison, is slightly more sophisticated. This is how
integration by parts generally reads:
given functions $u$ and
$v$,
$$
\int u \, dv = uv - \int v \, du
$$
While this is the form that high schoolers may be usually familiar with is, it also
helps to remember the more *symmetric* form of the equality above:
$$
uv = \int u \, dv + \int v \, du
$$
The basic idea behind integration by parts is this: it may happen that $\int u \, dv$
strikes no bells,
and might be harder to integrate directly (or by substitution), we might be able to
solve the easier $\int v \, du$.

The problem is to compute the (indefinite) integral $I = \int e^x \sin x \, dx$. Considering the form of the integrand, we are unable to directly apply substitution to the problem. Given this, it might make sense to try the other technique in our repertoire - integration by parts! Ok, so let's start.

Once we have decided to apply integration by parts, we realise that we have a couple of options here. Either $dv = e^x \, dx$ or $dv = \sin x\, dx$. In the former case, $v = e^x$, while in the latter $v = \cos x$. Let us pursue both of these possibilities as independent threads:

- $dv = e^x \, dx$, $u = \sin x$ gives $v = e^x$ and $du = \cos x\, dx$. So plugging this into the equation for integration by parts, we get: $$ I = \int u\, dv = e^x \sin x - \int e^x \cos x\, dx $$
- $dv = \sin x\, dx$, $u = e^x$ gives $v = -\cos x$ and $du = e^x\, dx$. Plugging these values into the equation for integration by parts again, we get: $$ I = \int u\, dv = - e^x \cos x - \int e^x (- \cos x)\, dx = -e^x \cos x + \int e^x \cos x \, dx $$

Ok, so now we have two expressions for $I$ and the *very interesting thing*
is that, in one of the expressions we have $\int e^x \cos x dx$ while in the other
we have the same integral with an *opposite* sign! So if we add the two expressions
for $I$, the pesky integral cancels away, and we get (and remembering to place the
arbitrary constant $c'$):
$$
2I = e^x (\sin x - \cos x) + c'
$$
which brings us to the solution:
$$
I = \frac{e^x (\sin x - \cos x)}{2} + c.
$$
Done. Finito. (No magic about $c'$ and $c$. We just wanted to reserve
c for the arbitrary constant in the last line; if you want, $c' = 2c$.)

Let's slow this process down. Suppose it hadn't struck us that we could
apply integration by parts *twice* and change the roles between $u$ and $v$,
what could we do then?

So, suppose we had applied integration by parts only once, and got
$$
I = \int u\, dv = e^x \sin x - \int e^x \cos x\, dx
$$
It looks that we are stumped here, since, if we applied another integration by
parts to solve/simplify $\int e^x \cos x\, dx$, we would *get back*
$\int e^x \sin x\, dx$ and no progress would be made.

Well at least let's compile the new information we do have. Let us pull the integral $\int e^x \cos x dx$ on the RHS over to the LHS and collect terms to get (along with the obligatory arbitrary constant $c$): $$ I_1 = \int e^x (\sin x + \cos x)\, dx = e^x \sin x + c $$

Note that in both of the solutions above, we had to realise that we can
apply integration by parts, and then we also had to apply it *twice*.
So in a sense, both of the solutions above are the same, just that the second
one is some sort of an *unrolling* of the first nifty-looking solution.

Given this,
let us call the solutions presented above as **Solution 1**, and the current
one as **Solution 2**.

Suppose it did not strike us to make that second application of integration by parts - what then?

Okay, let's take stock. It does need to strike that we will have to
apply integration by parts - *once*. But after that one application,
we are stymied; we have arrived at this equality:
$$
I_1 = \int e^x (\sin x + \cos x)\, dx = e^x \sin x + c
$$
And (suppose) we do not realise that we have to apply
integration by parts a second time (in a manner described above).

When we are stymied, its always good to look back and fiddle around with
what we already have. In this case, let us try to present what we have
*more succinctly*. To this end, let us try to understand the
expression $(\sin x + \cos x)$. What we mean by "understanding" here
is that, we would like to see this expression expressed as a *single*
sine or cosine (instead of both of them appearing). In other words,
we would like to "simplify" the expression containing both a sine
and a cosine into one that just has a sine (or a cosine).

At this point, we need to think back on the trigonometric identities that we
know. Where does this happen? Oh yeah, precisely in the expansion of
$\sin (A + B)$ (or $\cos (A + B)$). Recall the identity:
$$
\sin (A + B) = \sin A \cos B + \sin B \cos A
$$
This is clearly an example where lots of sines and cosines on the RHS are
*summarized * into a single sine on the LHS.

With this background, we may now manipulate our expression as follows: $$ (\sin x + \cos x) = \sqrt{2}\, \left(\frac{1}{\sqrt{2}}\sin x + \frac{1}{\sqrt{2}}\cos x\right) = \sqrt{2}\, \sin\left(x + \frac{\pi}{4}\right) $$ So we may rewrite the identity for $I_1$ as: $$ I_1 = \int e^x (\sin x + \cos x)\, dx = \sqrt{2}\, \int e^x \, \sin\left(x + \frac{\pi}{4}\right) \, dx = e^x \sin x + c $$

Let us extract out the essence of this last identity as follows: $$ I_4 = \int e^x \, \sin\left(x + \frac{\pi}{4}\right) \, dx = \frac{1}{\sqrt{2}}\, e^x \sin x + c $$ (where we are being sloppy about the arbitrary constant $c$, the $c$ for $I_4$ is a scaling factor away from the $c$ for $I_1$.

Personally I find this identity for $I_4$ rather beautiful. What this means is that
(modulo the scaling factors $\frac{1}{\sqrt{2}}$), integrating the LHS leads to
a *phase change* by $\frac{\pi}{4}$.

And now we can get one more solution to our original problem! Stare hard at
the last identity for $I_4$. The integrand there is quite close to the integrand
we want: $e^x \sin x$. So we realise that we do
not need *integration by parts* any more, but instead *integration by
substitution*! Let us proceed to substitute $t = (x + \frac{\pi}{4})$. In terms
of $t$, the identity above reads:
$$
I_4 = \int e^{\, t -\frac{\pi}{4}} \sin t \, dt = \frac{1}{\sqrt{2}} e^{\, t -\frac{\pi}{4}} \sin \left( t - \frac{\pi}{4} \right) + c
$$
and now cancel the $e^{-\frac{\pi}{4}}$ terms on either side, and we are done!

Given all of the above, there is a third solution that involves (a) neither integration by parts, nor (b) integration by substitution. This post is already getting way too long, and I will relegate the third solution to another post. The reader is invited to taste the teaser in the takeaways below.

- Solve the indefinite integral $\int e^x (\sin x + 2\cos x) \, dx$.
- Solve the indefinite integral $\int e^{2x}\sin x \, dx$.
- How about $\int e^x \sin^2 x\, dx$? $\int e^x \sin^3 x \,dx$? In general,
we can solve for $\int e^x \, q(\sin x) \, dx$ where $q(\cdot)$ is an
arbitrary polynomial. To stretch this, we can in fact solve for
$\int p(e^x) \, q(\sin x) \, dx$ where $p$ and $q$ are
*both*arbitrary polynomials!

- The main takeaway is that integration by parts is a powerful tool; if one application does not get you the result, consider using it a second time!
- Also, it may be useful to manipulate the expressions one already has in order to get extra insight about the problem. This is what we did in order to remove the extra application of integration by parts (in the second solution).
- I invite the reader to stare hard at the identity for $I_4$. Can there be a physical interpretation of this identity (maybe wandering into the area of Fourier series even)?
*The teaser:*I mentioned in the text above that there is another direct solution that does not use integration by parts. It uses! This will be the subject of another post.**complex numbers**

Created 13 July 2017.