Why is it so difficult to untangle a knot?

Short answer: If there were a simple formula to follow there would also be one for proving or disproving any mathematical statement.

Before setting out a sketch of a proof I need to be more clear about the problem.  I want to rule out the trivial method of untangling where you simply thread one end backwards through the tangle till it meets the other end.  So we're going to imagine it's a loop of string that's knotted, and "untangling" means getting back to a simple loop, without magically passing one bit through another.

For most closed loops it's not possible to untangle to a loop.  For example if you look at the Trefoil Knot above (a.k.a. Granny Knot) it's obvious that no amount of fiddling with it is going to get rid of the knotty bit and leave just a loop.  But if your loop $L$ can be deformed into a simple loop then it follows that the complement in 3D space $\mathbb{R}^3 - L$ can be deformed continuously into the complement of the simple loop $\{(x,y,z) : z \ne 0 \wedge x^2+y^2 \ne 1 \}$ implying that the two complement spaces are homeomorphic.

So we've found an invariant of equivalent knots: that their complement spaces are homeomorphic.  And any invariant of homeomorphic spaces must therefore give rise to an invariant of equivalent knots.  What we're going to do is look at one particular invariant known as the "fundamental group".  What's interesting about these is that they can be described in a completely abstract and algebraic manner.  When expressed in this way, the task of proving that two fundamental groups are equivalent is a case of manipulating symbols on paper using rules that indicate which sequences of symbols can be swapped out for which.  That's very similar to how you formally prove a theorem using axioms and rules of inference.

The fundamental group

A group is a very simple mathematical structure.  It consists of a pair of objects one of which is a non-empty set $G$ and the other a binary operation on that set $\cdot$. Normally $a\cdot b$ is shortened to simply $ab$.  The three axioms that $\cdot$ must satisfy for $(G,\cdot)$ to be considered a group are

1. Unity: there is an element $1$ s.t. $1a = a$ for all $a \in G$
2. Associativity: for any $a,b,c\in G$ we have $(ab)c = a(bc)$
3. Invertability: for any $a \in G$ there's an element $a^{-1}\in G$ such that $aa^{-1} = 1$

Given a connected real manifold - such as the complement of a knot - we can form a group by choosing an origin and looking at loops that both start and end there.  Strictly speaking the elements are not the paths themselves but equivalence classes where two paths are in the same equivalence class if one can be deformed into the other.  Thus the fundamental group of $\mathbb{R}^3$ is the trivial group because any closed loop from $\underline{0}$ to $\underline{0}$ can be deformed into any other.  Multiplication is achieved by simply joining any two representative paths together.

So far so good.  But the clever bit is finding a way to go from a picture of a knot, which is by nature continuous, to an algebraic representation that's more discrete.  We're going to think of the origin as above the paper on which the knot is drawn, and start off by labelling the continuous bits of the loop as seen from above. We'll also choose a direction around the knot:

As well as representing a bit of the diagram $a$ represents an element of the fundamental group.  It's the loop which goes around that part of the knot in the orientation indicated by the right hand rule (that's why we needed to add an arrow).  It's fairly intuitive that $a$,$b$, and $c$ also generate the group, which is to say that every element of the group can be obtained by combining them and their inverses.  Simply append $a$, $b$ or $c$ every time the path passes under the corresponding line in the direction indicated by the right hand rule, and  $a^{-1}$, $b^{-1}$, or $c^{-1}$ every time it passes under it in the other direction.  Close your eyes and think about why this works....

Specifying the generators is not enough on it's own to describe the group.  If it were then any knot with $n$ crossings would have the same fundamental group as the simple knot which can also be drawn with $n$ crossings.

In fact the crossings give rise to relations between the generators.

A simple way to memorize the relation is that you go clockwise around the crossing and append $x$ if you intersect $x$ and the arrow points into the crossing, and append $x^{-1}$ if you intersect $x$ and the arrow points out of the crossing.  The resulting expression is then equal to the identity.  Applying these rules to our Trefoil knot we get three relations

So our fundamental group is

$$
<a,b,c\space|\space bca^{-1}c^{-1} = 1, cab^{-1}a^{-1} = 1, abc^{-1}b^{-1} = 1>
$$

This is the set of equivalence classes of terms like $ba^3c^{-1}ab^{-2}a^3$ where equivalence means that you can get from one term to the other using just those three rules (plus the default ones which are always true: $x^mx^n = x^{m+n}$, and $x^0 =1$).

Equivalence of representations

The manner of describing a group used above is often referred to as a "representation".  It's easier to work with this description of a group than talking about continuous deformations of paths, as you can just do a bit of algebraic manipulation to show that two groups are the same.  For example, the three rules are equivalent to

$$
\begin{align}
bc &= ca \\
ca &= ab \\
ab &= bc \\
\end{align}
$$

We can ditch $bc = ca$ as it follows from the other rules, and rewrite the remaining rules as
$$
c = aba^{-1} = b^{-1}ab
$$

Which shows that $c$ is a redundant generator.  Thus the group can equally be represented as

$$
<a,b \space | \space aba^{-1} = b^{-1}ab>
$$

We've seen that demonstrating that two representations are the same involves taking a set of strings of symbols (e.g. $bca^{-1}c^{-1} = 1$), to produce other strings of symbols, using permissible rules (e.g. you can append the same expression to both sides and reduce expressions using group theory axioms).  This is very similar to how you formally prove theorems in logic: you start off with strings of symbols (the axioms), to produce other strings of symbols (the theorems), using permissible rules (i.e. logic axioms).  So the final step (and it's a biggy) is to produce a mapping between arbitrary mathematical statements and group representations such that one can show the group is the same as $<a>$ if and only if one can show the mathematical statement is true.  (One further complication is that the representations mapped to must be ones that can arise from knots.)

Once this has been done we can say with some confidence that we'll never find a method for untangling an arbitrary knot.  Why? Because, if we could then given any true mathematical statement we'd be able to find a corresponding knot, and  untangling it would prove that the statement was true.  But  Goedel proved there's no single method for proving the truth of an arbitrary true statement.

👋 QED (sort of, with much handwaving) 👋