We talked about this in an earlier thread about Mega STE & running games, since Mega STE and the "standard" STE essentially is the same when it comes to gaming... here's a recap.
There's basically 3 hurdles when it comes to compatibility issues with STE.
1. TOS. TOS 1.62 is basically the same as TOS 1.04, but stuff that relies on older TOS than that won't work. Unfortunately, the lesson was not completely learned by some ppl, so when going to TOS 2, some stuff won't work because they expect TOS 1.x
2. Different adressing of the PSG. They removed all shadowregisters of the PSG on the STE. Anything coded to use them, (to save a few clockcycles), won't work. <- This is the MAIN reason a lot of ST stuff doesn't work on STE. And this one is not easy to work around.
3. Memory. This is actually not specific to the STE, but since it is so easy to upgrade, ppl tend to think it is an STE problem. Without getting to technical, if the programmer was sloppy, he might write into memory locations just a few byte over memtop. Now, if memtop is lower than 4mb that's no big deal (ie, you got a 512kb, 1mb or 2mb machine). But if he crosses the magic 4mb boundary (which could happen on a 4mb machine), GLUE will issue a buserror, effectively crashing the computer.
Normally on ST(E), TOS will place the screen just below memtop. (on a 4mb machine that equals 4mb -32kb, since the screen occupies 32 kb.)
Sloppy programming in (mostly) older stuff, writes past the end of the screen, and if you cross 4mb with a write, the hardware get's upset.