Today I looked more into how I will deal with communication between the indexer, database, and the API. First I thought if simply having an HTTP server that in itself is an a way a REST API, but this is terribly inefficient as it (at least in my design) does one operation per request. So I started looking into other options. I asked around and was given the ideas of gRPC or Cap’n Proto. I honestly don’t entirely understand both of these and their different purposes, but at the end of the day I need to select some way to let the TypeScript API talk with one of possibly many different systems. For the time being, and given the research done, it seems that the easiest choice would be gRPC since it’s widely supported and has plenty of documentation. Cap’n Proto is definitely faster in decoding and sending, but gRPC also has some of its own benefits (again, this is no final list, nor am I deciding that I will continue to go with gRPC, but my initial program will use gRPC most likely)
- good documentation
- all platforms needed are supported
- easy enough to learn system
- fast enough and safe enough for my purposes
- uses multiple cores if needed
So, at this point it seems that some parts of the design are now complete. I now know that I’ll be doing a Redis-backed database for rankings and possibly user data, and continue to maintain elements of ElasticSearch and possibly moving to MongoDB for other pieces of data. The API will be in TypeScript, while the data “orchestration” system will likely be in Rust (open source, and a good opportunity to learn), while the indexer will likely also be TypeScript, unless I think I can give Go a try (doubtful, but will see with initial designs)
Sources
https://linuskarlsson.se/blog/grpc-between-rust-and-node.js/
https://github.com/hyperium/tonic
https://capnproto.org
https://news.ycombinator.com/item?id=14824477
https://blog.logrocket.com/rust-and-grpc-a-complete-guide/