We have had two good solutions to this problem. Both noticed that if you take any prime number greater than $3$, square it and subtract $1$ you always get a multiple of $24$.

Biren from The Heathland School went on to write:

I tried this with as many prime numbers as I could and it always worked. I then used algebra to try and prove it.

$p^2-1$is the same as $(p+1)(p-1)$. As $24$ was a large number to handle, I wrote it as the product of its prime factors ie. $3 \times2 \times2 \times2 = 3 \times 2^3$

Now we have a statement stating:
$(p+1)(p-1)$ is divisible by $3 \times2^3$.

We have to prove that $3$ will go into either $p+1$ or $p-1$ as this would then make $(p+1)(p-1)$ a multiple of $3$ which we want. Any prime number $p$ greater than $3$ when divided by $3$ can give two remainders: $1$ or $2$.

If $p$ left a remainder of $1$ when divided by $3$, then $p-1$ would leave a remainder of $0$, which would mean that it was a multiple of $3$. So therefore $(p+1)(p-1)$ would be divisible by $3$ as $p-1$ is divisible by $3$.

If $p$ left a remainder of $2$ when divided by $3$ then $p+1$ would leave a remainder of $0$, which would mean that it was a multiple of $3$, excellent! So therefore $(p+1)(p-1)$ would be divisible by $3$ as $p+1$ is divisible by $3$.

Now I have to prove that $2^3$ will go into $(p+1)(p-1)$. If a multiple of $n$ was multiplied by a multiple of $x$ then the answer would be a multiple of $nx$. (eg. Multiple of $4 \times$multiple of $7$ is a multiple of $28$ ie. $8 \times14 = 112$ which is divisible by $28$). Since $p$ is odd, $p+1$ is a multiple of $2$ and $p-1$ is the previous multiple of $2$, so $(p+1)(p-1)$ is a multiple of $8$, since either $p+1$ or $p-1$ is a multiple of $4$ and the other is a multiple of $2$ (if all the even numbers were written out: $2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,32 \ldots$ the first is a multiple of $2$, the next is a multiple of $4$, the next is a multiple of $2$, the next is a multiple of $4$, etc.)

And so since $(p+1)(p-1)$ is both a multiple of $8$ and $3$ it is also a multiple of $24$.

In a similar vein, Ning from Raffles Institution in Singapore wrote:

I notice that the numbers are always divisible by $24$.
To prove that this will go on infinitely, I show that the numbers are always divisible by $8$ and $3$, which means that they will be divisible by $24$.

As the starting number is a prime greater than $3$, it must be $2$(modulo $3$) or $1$(modulo $3$). The square of the prime number is always $4$(modulo $3$) or $1$(modulo $3$). Therefore, the square of the prime number $- 1$ is divisible by $3$. The prime number at first is either $1$(modulo $8$), $3$(modulo $8$), $5$(modulo $8$), or $7$(modulo $8$), as it is odd. The square of a $1$(modulo $8$) is still $1$(modulo $8$), the square of a $3$(modulo $8$) is $9$(modulo $8$), or $1$(modulo $8$), the square of a $5$(modulo $8$) is $25$(modulo $8$), or $1$(modulo $8$), and the square of a $7$(modulo $8$) is $49$(modulo $8$), or $1$(modulo $8$). Therefore, the square of the prime number - $1$ is divisible by $8$.

Therefore, the final numbers would be divisible by $24$ to infinity.

Thankyou both for such good solutions.

Well done to all those people who submitted solutions. Jared, from Jenks High School, made a conjecture that dividing by 24 would always give a whole number, and tested it for the first few primes. Tomas from Malmesbury School, Anurag from Queen Elizabeth's Grammar School, and Alex and Jason from Colyton Grammar School all went further and proved their conjecture. Here is Alex and Jason's proof.

Squaring a prime number, subtracting $1$ and dividing by $24$ can be written as
$${p^2-1}\over{24}$$
This expression can be factorised as
$${(p+1)(p-1)}\over{24}$$

So we need to prove that $(p+1)(p-1)$ has a factor of $24$

To do this we split $24$ into its prime factors, $2^3 \times 3$

$p$ is prime and so can't be a multiple of $3$, so either $p-1$ or $p+1$ must be a multiple of $3$.

We also know that $p$ cannot be a multiple of $2$, so $p-1$ and $p+1$ must both be multiples of $2$.
We know that alternate multiples of $2$ are also multiples of $4$, so either $p-1$ or $p+1$ is a multiple of $4$, giving us our third factor of $2$.

Therefore $(p+1)(p-1)$ has a factor of $2 \times 2 \times 2 \times 3$ which is $24$.