back
99 comments
I'm doing similar, it's so much fun, and here are my takeaways:

- Ignore socket.io, do pure binary websocket

- Do not serialize JSON, or use msgpack or protobuf - write your own protocol.

- Write tests for protocol, you need fast iteration. It's super easy to write tests for codec.

- Quadtrees are kinda lousy compared to r-trees.

- Measure everything with realistic load - i.e. I've found out that it's more costly to calculate world entity deltas to send to client, than to send everything in his view.

- Keep client updates under MTU payload. This is basically impossible with JSON.

- Mutability is the way to go for performance.

- Share code with backend and frontend, you'll need it for client prediction.

- Prefer fixed entity speeds.

- Don't log per-frame in production, journalctl will eat up your memory.

- Write game bots to test your load non-synthetically.

- Test on your VPS, you'll be surprised how underperforming CPUs are in most providers. Neighbours are probably serving some CRUD and are very likely not to be as noisy as you are.

Now, all this might not apply depending on exact gameplay mechanics you have, but it works for me. I have Node.js serving 200 clients and 10k entities with vision (can run at or from you) at 20 frames per second that keeps under 100MB RAM with no hiccups. I think I could use something like Golang for some more advanced physics though.

A question to all reading - I'm considering adding an option to mine Monero in my next game. It'd probably be a flip-switch between ads and mining right there on the home screen, with some intelligent throttle/thread decisions depending on the kind of device game is running on. I'd appreciate any feedback regarding the sentiment on that.

Those are some very good points, but I think some of them are wrong or do not apply for most uses cases.

1) "Do not serialize JSON, or use msgpack or protobuf - write your own protocol." - Why should you write your own protocol insead of protobuf? If your game has more than 10-20 types of messages/objects than it becomes very hard to implement your own binary encoding for each object. Plus with protobuf is easy to use and the packets generated are really smaller (most likely smaller than your own protocol).

2) "Measure everything with realistic load - i.e. I've found out that it's more costly to calculate world entity deltas to send to client, than to send everything in his view." - Profiling is something that you should always do, with both synthetic and real loads. In your case, my guess is that your diffing code was inefficient or that the load was so small that both computing the delta and sending the message were almost instant, so comparing them would give inaccurate results.

3) "Prefer fixed entity speeds." - I think this is a game design decision and you shouldn't limit your game to this if you don't have too. Also, I don't think there's any issue with entities being able to have a different speed each tick.

And, a question. What VPS are you using? Do you recommend any specific VPS for hosting Node.js game servers?

1) Demonstrably not so, in my case. Devising and implementing a protocol flexible enough for me was, along with test-driven implementation of that particular aspect, a joy and more performant solution than any other out of the box. Yes, I've tested against msgpack/protobuf (and I use protobuf in production on other projects) and found it to be less optimal, for both implementation (you're never as flexible as with blank board) and runtime resource usage, across board.

2) Sure, I might go the way of the delta one of these days - it does still smell like good design. That's why I was underwhelmed to see it underperform first time around.

3) Yeah, design and implementation are never separate. Neither is content and presentation. :)

DigitalOcean works for me now. Wouldn't mind slightly better CPUs, as even High Compute droplets are not all they're cracked to be.

>>> A question to all reading - I'm considering adding an option to mine Monero in my next game. It'd probably be a flip-switch between ads and mining right there on the home screen, with some intelligent throttle/thread decisions depending on the kind of device game is running on. I'd appreciate any feedback regarding the sentiment on that.

Mining => You WILL murder the battery of your users. You WILL cause overheating of the devices. You will also grind the browser to a halt on all hardware of moderate speed or already busy for reading a youtube video in the background.

Personally. I'd consider that any website that start mining when opened should go right into an adblock list, if not blocked as malicious entirely.

I'd suggest finding out, in the real world, if it is ads or mining that gets you the greater income. Then, you optimize for that.

In either case, when you're having them doing the mining, I'd recommend full disclosure AND I'd recommend some explanatory text that goes into the details of what you're doing and why.

Something like, "Click here to learn more about mining." That should do the trick. I'd just put said link below the quick blurb/switch/invite to use mining.

If mining generates more income, then you may wish to offer some sort of in-game bonus for those who enable it. You'll have to figure out some way to ensure they really have it enabled, of course. Giving them a reward may entice more users to choose to do mining.

Another post mentioned that this would be a battery killer. You responded that you'd only enable it for certain devices/browsers. If you want, you could probably enable it (as a choice) for all devices and browsers BUT I'd absolutely ensure that you made it very clear that doing so would mean a rapid depletion of stored energy on battery powered devices.

The reason I'd suggest you do allow it is because many people only use mobile devices and, in some cases, will operate them while they are plugged in. For example, I'm typing this on a tablet that is plugged into an outlet. If mining makes you income, I'd be inclined to want to do mining for you - even while on a mobile device. It'd be even more tempting if there were an in-game reward for it.

Sort of related: That might make an interesting funding option. Free games but mandatory mining on behalf of the creator. Obviously, full disclosure is required if one is concerned with ethics. Inside this idea, they might be able to get rewards directly related to how much they mine on the creator's behalf.

>>> I'm doing similar, it's so much fun

Yes, basically this. I think most of us solve the same sort of problems day in and day out with our jobs; it's unbelievably fun to be dropped into a totally new problem space and have that feeling of complete ignorance again.

Nice list! These are pretty useful. I found the experience of using protobuf for frontend <-> Go communication to be good, since it forced me to do the work of 'writing my own protocol' but without all the encoding/decoding. There was some issue with naming in protobuf and Go's conventions though.

To add my 2 cents: I think mining Monero is actually very reasonable (you seem to have thought through the impact on your users), and you've nailed the most important thing: making it opt-in.

I'm fine contributing CPU cycles when playing a game, as long as it's in the foreground and I agreed to it.

> I have Node.js serving 200 clients and 10k entities with vision (can run at or from you) at 20 frames per second that keeps under 100MB RAM with no hiccups.

What's in the 100MB?

Doing some napkin math, take 10k hypothetical game objects that each have as state: pos, vel, acc, rot, ang vel, ang acc, name (32 chars), color, health, mana, armor, level, type, energy, kills, timers, inventory (256 bytes)... etc etc, all together comes in at around 512 bytes each. Even if that's double buffered it's still just 10MB.

Curious where all that memory is going!

Please tell me what your game is. Even considering abusing my hardware for mining is enough reason for me to avoid anything you create.
You say "Ignore socket.io, do pure binary websocket" and that sort of makes sense to me, along with your advice about not using JSON, but what are you using to do "pure binary websocket"?

Do you have any code samples, please? (Don't need to run or compile, just an example would be handy.)

Also, can you post a link to your game, please? I'd like to give it a try.

There are so many msgpack and protobuff implantations on npm, does anyone have recommendations on which to use / what the tradeoffs are?

https://npmcharts.com/compare/avsc,thrift,msgpack-lite,msgpa...

> I've found out that it's more costly to calculate world entity deltas to send to client, than to send everything in his view.

If you don't mind me asking, Why is this the case? I would have assumed world entity deltas is probably how I'd approach things first...

>Write game bots to test your load non-synthetically.

Can you expand a bit more on that? Cool tips, for sure. Personally I'd appreciate a flip-switch between ads and mining, as long as it defaulted to ads.

Are you sure about no JSON? My experience is JSON is fairly small, and faster than anything else because it has browser support.
Having worked on a commercial networked first person shooter, I can verify that making it smooth and coordinated is a massive pain in the ass. Having a game like World Of Tanks where the server makes all the decisions only works if the game entities are big and don't change much (i.e. tanks). There is a reason why none of the World games have people. Nothing is harder than trying to make a knife fight work over the internet without it looking spastic.
It's also why many MMORPGs rarely try to do collision testing, and have gameplay mechanics where you queue up abilities in advance of expecting to see their effects.
I'm working on a project I hope to make multiplayer eventually, but the issues surrounding it scares me a bit. I wish more companies doing the hard multiplayer work (fps's mostly) would reveal more of their back end and how they do things.

I don't think anyone has really done it well yet but everyone is hiding what they have figured out so far. I have noticed that certain engines tend to have better success at this, for example source engine seems to have drastically improved, allowing games working on this issue like Insurgency to make some noticeable strides towards being better.

This has me remembering UT over dialup lag...

Don't not share - which FPS? :)

Yeah, game design and tech to support it are tightly linked, and there's approaches to it from either side.

"spastic" Are there not better words ? Uncoordinated ? Glitchy ? clumsy ? awkward? blundering?
Just wondering, why are people still using socket.io and not standard websockets, as available in every browser since several years now?
If you don't mind performance, using socket.io gives you stay-alive on the ws connection. If you don't mind writing the 5 lines of code it takes to implement stay-alive for normal websockets then socket.io makes no sense anymore.
I'm personally using it because it gives me:

1. Rooms

2. Disconnects and reconnects,

3. Fallack solutions

for free.

Although not for a game. I trust that its just not made for that kind of performance.

But for a messaging app its actually quite decent.

It provides fallback connection methods if WS connection fails, as it can do if you have, for example, a HTTP/1.0 only proxy between you and the client. For example it was not that long ago that nginx did not support proxying HTTP/1.1. It is getting less and less relevant but at least for me the socket.io code has worked for a long time and there hasn't been a need to rewrite it.
In case you haven't reached that point yet : also evaluate how much network badnwidth is going to cost you if you ever host your servers on aws or gcloud and encounter average success. You may be in for a (bad) surprise.

Now compare that to how much you'll gain from ads. And soon you'll realize one huge issue you're going to have is to maintain your own servers a little bit everywhere in the world.

I've carefully measured and worked toward 1TB/month/VPS at full load. Didn't get there, but now I'm at 2TB (still estimates). With socket.io and JSON, that'd easily blow up to few hundreds TB/month. DigitalOcean for now (but that might change in the future) still doesn't charge on overages, but you do get a slap on the wrist of some kind if you go way overboard, so it's preferable solution to other VPS providers.

It's good that you mention this, since I've heard "horror" stories of people working in this space earning about $10k/mo but spending as much and more on AWS.

This is a great write up. I've had similar experiences, and agree, have the server-side game state be the single source of truth.

I basically stuck some visuals on top of the example "counting" snippet on the Elm site (http://elm-lang.org/examples/buttons) and made it multiplayer. First player to count to 50 wins.

I didn't use a tick approach, instead the state of the game was only updated on an event sent from one of the clients. I then broadcast the new state of the game to the other clients.

This is what I made http://retrorace.neillyons.io/ https://github.com/nwjlyons/retrorace. Something is not quite right in the backend code. I think there are problems with waiting for the mutex to unlock. The game feels like it freezes sometimes, but it could also just be a latency issue. The server is in London and I've only tested it with people who are several hundred miles away.

I also had plans to build multiplayer snake but felt like latency would make the game too slow.

I try to squeeze a little Go side-wise, so I'll be sure to check this out in the future. Thanks.
If you have problems with lag and conflicts resolution of state, take a look at this article, it may be useful. https://0fps.net/2014/02/17/replication-in-networked-games-l...
On the topic of multiplayer HTML5 games, I've recently started playing with Lance[1] and its pretty good. Its main draw is that it largely does the networking and latency compensation (it has options for interpolation and extrapolation) for you. The only requirement is that you write your game (client and server) in javascript and run the server on node, so that the gameplay logic can be run both in server and client (game is simulated both on client and server, but the clients are periodically synced with the server, using interpolation to make it look smooth).

[1] http://lance.gg

If you are writing a game server with web, mobile and desktop clients, is socket.io the way to go?

AFAIK, socket.io is a wrapper around websockets. This makes it unsuitable for some fast paced games with alot of players. Is there a library which abstracts away tcp and udp connections?

And what can you do about tcp attack vectors? Are lamers trying to take down servers a real threat? How do you prepare for this? I therefore advise against writing your own network stack if you intend to make the service available to the public. Or am I too cautious?

> If you are writing a game server with web, mobile and desktop clients, is socket.io the way to go?

Socket.io's most discerning feature was fallback to polling. There's also some useful abstractions there, like reconnecting, rooms, as well as some useful ecosystem around it, like socket.io-redis adapter which enables you transparently scale horizontally. If your time is at premium, you might find socket.io work for you, but it's not really hard to write everything better yourself, and better tuned to your use-case.

> AFAIK, socket.io is a wrapper around websockets. This makes it unsuitable for some fast paced games with alot of players.

For fast-action multiplayer, there's currently no sensible offering besides WebRTC's DataChannel, with all it brings on board (I've yet to study it in-depth). I'm having high hopes for QUIC to gain more ground.

> And what can you do about tcp attack vectors? Are lamers trying to take down servers a real threat?

From what I've experienced first hand, bots/scripts are most frequent enjoyment-spoiler for other players. I've never ran at scale where I'd be interesting target for actual attacks.

As for udp connections, WebRTC looks like the only mature solution (but it's not designed for games and is way more complicated than game devs want it to be). Alternatively, I know about https://github.com/networkprotocol/netcode.io but it's very much in the early stages - afaik you even need a Chrome extension to use it on the client. It's not easy to do udp on the web.
https://github.com/tinspin/fuse

TLDR: Websockets are overengineered and waste CPU cycles doing nothing but marshalling. HTML is better than canvas because it can be GPU accelerated and you get components and z-order for free.

Any not so expensive hosting options for sockets-based apps? Heroku? Digital ocean? AWS?

Also, what are the limitations by the hosting provider regarding concurrent connections, data transfer, etc.

I am currently working on a poker app using free Heroku dynos but would like to have a budget in mind for when going live.

I host https://babbl.xyz on DO with a single $10 server for the socket and the web, and a $10 server for mysql. This can support around 500 active players.

I also did a more action asteroids clone (http://triangle-wars.com), and this would only support I guess 50 players on a $10 server.

The limiting factor seems to be the number of packets / sec that need to be broadcast to all players. If each player is changing state 2/sec, and you have 50 players, then the sever needs to relay 100 packets/sec...

Unlike some other comments in this thread, I didn't find JSON parsing to have any effect on performance.

On DigitalOcean, I'm planning a following setup to launch:

1. $20 staging at all times, already live [1]

2. $20 static nginx

3. 3x$5 for socket servers on North America / Europe / Asia

4. $5 master server with WS connections to full mesh of socket servers / orchestration / future.

All Debian.

Have no clue what's Heroku like for that, but last time I've checked, it wasn't as good choice as DO was and is for me.

[1] WARNING: might break at any time, by choice or not http://staging.munchies.io

If you want to trade off simplicity for lower costs or lower latency, you can also use WebRTC data channels that do p2p communication between the players.
The author mentioned that using a control scheme with small acceleration leads to better synchronizing with the server. Another approach to this is control schemes where the client indicates to the server "I'm planning on being at this location at this future time."
That's then, logically, open to exploitation from user and client-side. Unless deeper logic layers are made to prevent that.

Rabbit-hole, I tell ya. They're fun, though. :)

I wrote something similar as a very simple demo multi-player tank game/simulator with socket.io and canvas: https://github.com/pdfernhout/TanksInYourBrowser
Do you have a live demo somewhere?
Great fun - one suggestion would be if it could tell you who you are when you start.
Good idea!
"io.call('up')" broke your server. Sorry :(
Just woke up and it seems to be working now. No harm done, I guess?