For running Java, you will need a J2ME loader as CE isnt able to do it out of the box (and neither is Windows Mobile for that matter). Im currently using JBlend (was cooked into the ROM im running, and runs pretty well) in Windows Mobile 6.5.5 (Windows CE 5.2, I used to use phoneME for a while, but my favourite when I had my HTC Universal was EsmertecJbed.
I take it the Opera Mini cab for Windows Mobile (no Java required) isnt running/installing correctly in CE?
A lot of mobile apps written for the QT framwork will run under Windows CE. Flashlite used to run under Windows CE, which enables you to view flash video's etc, however having said that, ive never found it to ive the most satisfactory results on my phones (sometimes takes longer than expected to buffer, sometimes the buffer is too large for the RAM etc), and its based on Flash7 so does not support the latest developments - though it was working on the iPlayer for example, last time I tried in Internet Explorer Mobile.
Native Flash 10 support for Windows CE/Windows Mobile was dropped last year when Microsoft announced the end of the WinMo platform - Adobe ceased development of 10.1 for WinMo. Development of Flash for WinCE 7 is happening as we speak, but this is likely to be limited to WP7 and thus almost certainly not compatible with earlier versions of CE, and probably even non WP7 devices.
Once you get used to using it, you will probably find that CE can be quite a powerful tool - the only problem you may come across, is a lot of the apps designed for it, are out of date now. The reason being of course, that though Windows Mobile and PocketPC both run on the CE 5 kernel, they are not directly CE, and thus a lot of the apps targetted at WinMo/PPC may not be able to run/stable within the core CE enviroment. Developers tend to use CE as the base, and then tweak it for the specific platform they are developing the application for. Though it is relatively straightforward for a developer to make a port between CE devices, the usual barrier is whether it is worth them bothering. On occasion though, you will get lucky and a PPC/WM app will run straight, or there will be an app that a developer has continued to code for core CE.
Best thing to do is just try installing cabs developed for PPC and WinMo - the worst that can happen is they will give you an error during installation, or at worst, simply fail to load/run correctly.
One aspect where many WinMo apps encounter problems (not so much on modern apps) is the screen resolution. When I had my HTC Universal for instance, because it had a 640x480 res screen, whilst being amazing to use, had serious problems running certain apps such as Windows Media Player etc, because they were not developed to support such a high resolution/that resolution in particular. Nowadays, my present phone is 800x480, but in addition to this, WinMo 6.5.5 supports 320x320, 176x220, 240x320 and id hazard a guess, most likely 640x480. In some cases, a res-tweak can be applied to make the apps run better, but if your device has one of these res's then you have more of a chance of running WinMo apps.
Unfortunately, no WP7 apps will run under any other CE based enviroment, due to MS restricting them to running only on the .net framework, so no WP7 apps are C++ - im guessing this is for QoS reasons, as it should make 3rd party apps far more stable and better developed.
Some good forums are the likes of
www.mobilism.org (formerly ppcwarez.org),
www.ppcgeeks.com and you may even find some good info and/or apps that are CE compatible on
www.xda-developers.com - though I must admit, gradually all these forums are moving towards Android and Windows Phone 7.
Speaking of which, my days with WinCE are likely numbered now, as im dual-booting my HTC TP2 with Android Froyo alongside WinMo. I wouldnt be overly surprised if someone somewhere hadnt developed a compatible Android release for your CE device - seems to be the way the world of CE is going now (despite MS [and now Nokia's] best attempts at pushing the beautiful, but seriously restrictive WP7).