As far as I can see, there 3 major classes of undocumented opcodes.
The first one is the classic 1970s of incomplete decoding, where there is no effort made to inhibit the execution of undefined instruction bit patterns.
E.g., on the 6502, instructions are generally laid out in a aaa.bbb.cc bit pattern, which pretty much defines the instruction grid. Generally, there are no instructions defined whatsoever, where both bits of c are set. What happens is that both the instructions for c=1 and c=2 execute at once.
E.g.:
inst aaa.bbb cc opc addr-mode (comment)
------------------------------------------------------------
$99: 100.110.01 … STA absolute,Y (store A)
$9A: 100.110.10 … TXS implied (transfer X to stack pointer [SP])
------------------------------------------------------------
$9B: 100.110.11 … TAS absolute,Y (A and X are transferred to the internal latch,
at once, resulting in `A AND X`,
which is then transferred to SP [as in TXS].
This is also stored at the provided address,
like STA, here an absolute address indexed by Y.
In the course of these address calculations,
the high-byte of the address [AH] + 1 is added to `A AND X`,
resulting in `(A AND X) + (AH+1)` being stored.)
I guess, we may see how unintentional this behavior is, but also, how this corresponds to the official instructions that are executed. This is classic undefined behavior.
There are also a few other cases, where there are “holes” in the populated parts of the instruction grid, where, what seems to be defined by the general decoding matrix, may either be without external effect or will fail entirely with the CPU becoming stuck. (E.g., a code may indicate that this is a STA instruction, but also with immediate address mode, where we read a literal byte value as the operand. As there is never a write address asserted on the bus, this will result in a NOP.)
The CMOS variants (65C02) add circuitry required to inhibit any effects. (While all undocumented instruction codes are now NOPs, they are still of varying byte length and execute with varying cycle counts.)
The second class is more modern and probably the most dangerous one: Over time, what are official instructions on the outside (to the programmer) and what is executed internally, drifted apart. We’re now dealing with two instruction sets: the official, external one and a mostly undisclosed internal one. And there may be undocumented ways to directly inject an internal instruction from the outside, or to transfer firmware code… Notably, these CPUs typically incorporate a security model (rings) of one way or the other, and accessing the internal instruction set may allow to break out of this guard rails.
And then there’s the amazing case of the 8085, a business decision, as described by @pdxjjb.