report: add exec sum section

This commit is contained in:
Bruno BELANYI 2021-08-04 19:12:54 +02:00
parent 53f34563bc
commit 0577e1d911
1 changed files with 90 additions and 0 deletions

View File

@ -1,5 +1,95 @@
# Exec Sum
When making my decision to major in Image Processing and Image Synthesis, my
main motivator was my growing curiosity for high performance programming, as I
thought that some of the subjects being taught would lend themselves well to
furthering my understanding of this domain.\
On my own, I discovered through a number of conference talks that I watched that
the field of finance provides many interesting challenges that aligned with
those interests.
When the opportunity arose to work for IMC, a leading firm in the world of
*market-making*, I jumped at the chance to apply to them for my internship. When
asked for the kind of work I wanted to do during that time, I highlighted my
interests in performance, which lead to the subject of writing a benchmark
framework for their new exchange connectivity layer, currently in the process of
being created and deployed.\
This felt like the perfect subject to learn more about finance, a field I had
been interested for some time at that point, and get exposed to people that are
knowledgeable, to cultivate my budding interest in the field.
During my internship, I was part of the *Global Execution* team at IMC, a new
team that was tasked with working on projects of relevance to multiple desks
(each of whom is dedicated to working on a specific exchange). Their first
contribution being to write the *Execution Gateway*, a piece of software that
should act as the intermediary between IMC's trading algorithm and external
exchanges. This is meant to replace the current pattern of each auto-trader
connecting directly to the exchanges to place orders, migrating instead to a
central piece of infrastructure to do the translation between internal IMC
protocols and decision, and exchange-facing requests.
As part of this migration process, my mission was to work on providing a simple
framework that could be used to measure the performance of such a gateway. To do
so, we must be able to instrument one under various scenarios meant to mirror
real-life conditions, or exercise edge-cases.
This lead me to first get acquainted with the components that go into running
the gateway, and what is necessary on the client side to make use of it through
the *Execution API*, which is in the interface exposed to downstream consumers
of the gateway: the trading algorithms.\
Similarly, I learned what the gateway itself needs to be able to communicate
with an exchange, as I would also need to control that side in the framework to
allow measurements of the time taken for an order from the client to the
exchange.\
To learn about those dependencies, I studied the existing tests for the
*Execution API*, which bundle client, gateway, and exchange in a single process
to test their correct behaviour.
Once I got more familiar with each piece of the puzzle, I got to work on writing
the framework itself. This required writing a few modules that would provide
data needed as input by the gateway as a pre-requisite to being able to run
anything. With those done, getting every component running properly was the next
step. With that done, the last part was to generate some dummy load on the
gateway, and collecting its performance measurements. Those could be analyzed
offline, in a more exploratory manner.\
With that work done, I had delivered my final product relating to my internship
subject. Taken from a first *Proof-Of-Concept* to a working end result. Along
the way I learned about the specific needs relating to the usage of those
benchmarks, and integrated them in the greater IMC ecosystem.
With the initial work on the benchmark framework done, my next area of focus was
to add compatibility testing for the gateway. The point of this work was to add,
as part of the *Continuous Integration* pipeline in use at IMC, tests that would
exercise the actual gateway binaries used in production, to test for both
forward and backward compatibility. This work fits perfectly as a continuation
of what I did to write the benchmark framework: both need to run the gateway and
generate specific scenarios to test its behaviour. This allowed me to reuse most
of the code what I had written for the benchmark, and apply it to writing the
tests.\
The need for reliable tests meant that I had to do a lot of ground work to
ensure that they were not flaky, this is probably the part that took longest in
the process, with some deep investigations to understand some subtle bugs and
behaviours that were exposed by the new tests I was attempting to integrate.
Towards the end of my internship, I presented the work I did on the framework to
other developers in the execution teams. This was in part to showcase the work
being done by the *Global Execution* team, and to participate in the regular
knowledge sharing that happens at IMC.
Joining a company during the COVID period of quarantines, working-from-home, and
the relatively low amount of face-to-face contact high-lighted the need for
efficient ways of communicating with my colleagues. Being part of a productive,
highly driven team has been a pleasure.
Overall, this internship was a great reaffirmation of my career choices. This
was my first experience with an actual project dealing with performance of
something actually used in production, and not simply a one-off project to work
on and leave aside. Being able to contribute something which will be used
long-term, and have impactful consequences, reassures me in the choice of being
a software engineer, and the impact of my studies at EPITA.
\newpage
# Thanks and acknowledgements
First off, I would like to thank Jelle Wissink, an engineer from the Global