The Zeo is still the best consumer product for REM sleep detection out there. If you think about getting a used one, I suggest paying a bit extra for the older Bedside version because there is a firmware for the docking station that also outputs raw data over a serial port. But if you really only need sleep stage detection, the newer Mobile (bluetooth) version works too.
The Zeo uses one EEG channel and an accelerometer, but the EEG channel also picks up eye movements if the headband is not placed too high. So there are three things to be analysed by the software.
I've tried before to do automated REM sleep detection with a single EEG channel using a ThinkGear toy EEG on a headband. But it never got reliable enough. The lucidscribe ThinkGear plugin (which uses only eye movements) works just as well, after a bit of tuning of the threshold setting. A good week of research and study may be enough to get a sleep stage detection as good as the one of the Zeo, but it won't come close to the reliability of trained personnel of a sleep lab. If anybody wants to have a try, I could provide ZEO recordings of many nights, including raw data. My first step would be to imitate the algorithm of the Zeo, then to port it to another device.
But since your question seems to be more about hardware, here are a few thoughts on this. You could just get a Zeo and use that, like almost everybody does. It works, and not too bad. But I think it is time for a successor. Of the not yet freely available consumer devices, I think Aurora sleep mask would be best for sleep related applications. But since this isn't available yet, the Melon headband might be an option. It's pretty light and it's surprisingly easy to forget that you are wearing it, given that your head happens to match one of the provided spacer sizes. The Melon has 2 EEG channels that could also be used to detect relevant eye movements and it has a 6-axis accelerometer.
Getting a bit more DIY, there is also the Halograph headband by Michael Paul, made for exactly this use case. But you'd have to be okay with wires. Different electrode placements might be useful, especially if you have 2 or 4 channels, so you could record from the forehead, including eye movements, and at the top, without eye movements.
Depending on how much wiring in/around the face is acceptable, you could of course also make it a bit more professional and record EEG, EOG and EMG. But I would focus on an easy to put on and comfortable to wear headband. OpenBCI would be in line for both alternatives.
There is also a headband-compatible version of the Brain-Duino, devided into three small PCBs instead of one big Arduino shield. But it's not available as a finished device (yet?), only kits (ask me for a contact if you are interested).
A very promising project may be the Dream Writer device ("Traumschreiber") that is currently developed by Kristoffer Appel and Johannes Leugeri at Osnabrück University. Kristoffer Appel has done dream communication research with a Zeo in the past. Their new device is mainly supposed to be a replacement for the Zeo for further research with study participants at home, keeping up the "sleep lab in a headband" image of the Zeo. There seems to be no information available on the web about this project. But I hope to get an idea of how the first prototype will look in a few weeks.
That's the options that I have in mind for automated sleep stage detection. If wires are acceptable, you could use almost anything. If low reliability is also acceptable, you could really use anything. It depends mainly on what you think is acceptable to wear during sleep and what you want to write software for.
Please keep us updated! I would love to see anything new in this, no matter how DIY.