This was the subject of a fascinating talk at FOSDEM in 2018: https://archive.fosdem.org/2018/schedule/event/dosemu_and_fr...

The slides are here: https://archive.fosdem.org/2018/schedule/event/dosemu_and_fr...

TL;DR summary...

The original DOSemu uses the x86-32 V86 mode, as discussed in this HN post today: https://news.ycombinator.com/item?id=24555689

Essentially, the 386 had a sort of hardware-assisted VM that let it run 8086 OSes under multiple virtual 8086 machines. https://en.wikipedia.org/wiki/Virtual_8086_mode

The problem is that x86-64 doesn't have this any more.

Slightly more info: https://dosemu2.github.io/dosemu2/