Simple HTTP client using proxygen
This article is about another piece of technology that is open sourced by Meta. It is called proxygen. Initially framework was envisaged as a collection of tools used to develop proxies. Hence proxygen. However, this library is so much more. It is a library with a number of abstractions that can be used to develop any kind of client or server.
In this article we will go through developing the most simple HTTP client possible. There are a lot of moving pieces so I’d like to keep complexity to minimum and only focus on what is important. If you’re reading this, you probably have a good head on your shoulders and you will be able to expand on this.
First let’s start with very general proxygen architecture. Data is either read from a socket or written to a socket. We then have HTTPSession that uses HTTPCodec to read/write data to/from the socket. Each HTTPSession will have one or more HTTPTransactions. Each HTTPTransaction interacts with a Handler as data is received/sent. You can see this illustrated in the image below.
This very simple, yet powerful set of abstractions allows handling all sorts of other protocols like HTTP2, HTTP3/QUIC/SPDY including own “home” grown protocols.
Before you start working with any of these you’ll probably want to build most basic application to make sure the project is set up correctly. I personally found this to be the most challenging part. Even though proxygen comes with it’s own build script which ensures that examples build and can be run, I didn’t find this the most optimal way to start a new project. Porxygen relies heavily on some other libraries (e.g. folly, libevent, glog, gflags…) so you will need to make sure that those are present on your system. You might also want to customize which compiler and/or linker is used. I personally use clang compiler and mold linker.
In order to help with the above problems, I created a repository that does the following:
introduces most dependencies as git submodules
provides a build script to build all dependences
implements a simple HTTP client (KissClient, Kiss stands for keep it stupid simple)
provides instructions how to build the client
Caveat: the project currently only works on linux. At the moment I don’t have plans to make it work on MacOS X or Windows. Having said that, I am open to accepting a pull request.
This repository should be enough for you to fork and go of make your own project based on proxygen.
If you run into any problems, please open an issue (https://gitlab.com/technologysoup/starter-kits/hello-proxygen/-/issues) or reach out via discord (https://discord.gg/vFbZFzMmDK)
KissClient class extends two proxygen classes - proxygen::HTTPConnector::Callback and proxygen::HTTPTransactionHandler.
proxygen::HTTPConnector is used to establish new HTTP or HTTPS connections. The proxygen::HTTPConnector::Callback defines callback object which will receive results. The callback object has connectSuccess and connectError methods.
HTTPTransaction is another object of interest. It represents one request/response pair in an HTTP-like protocol. It works with a Transport and a Handler to handle ingress and egress of data.
Normally the sequence of events is:
The application creates a transport object. This object will create a transaction to represent the request.
Once the transport has received full headers it will create a handler object and tie it to the transaction.
When the handler is ready to produce response it streams it through the transaction to the transport.
Eventually handler and transaction will “go out of existence” and transport is free to process another request.
In the KissClient example, the handler is of proxygen::HTTPTransactionHandler type.
In the next few articles I’ll be showing how to do something more complex using event loops.
If there is something specific that you’d like to learn more about please let me know in the comments below.
Lastly, if you learned something useful please share this post and consider subscribing to get notified when my next article comes out. I try to publish on a weekly / bi-weely basis - depending on the topic and amount of “free” time I have.