report: add exec sum section

2021-08-04 19:12:54 +02:00 · 2021-08-04 19:12:54 +02:00 · 0577e1d911
parent 53f34563bc
commit 0577e1d911
1 changed files with 90 additions and 0 deletions
--- a/report.md
+++ b/report.md
@ -1,5 +1,95 @@
 # Exec Sum
 When making my decision to major in Image Processing and Image Synthesis, my
 main motivator was my growing curiosity for high performance programming, as I
 thought that some of the subjects being taught would lend themselves well to
 furthering my understanding of this domain.\
 On my own, I discovered through a number of conference talks that I watched that
 the field of finance provides many interesting challenges that aligned with
 those interests.
 When the opportunity arose to work for IMC, a leading firm in the world of
 *market-making*, I jumped at the chance to apply to them for my internship. When
 asked for the kind of work I wanted to do during that time, I highlighted my
 interests in performance, which lead to the subject of writing a benchmark
 framework for their new exchange connectivity layer, currently in the process of
 being created and deployed.\
 This felt like the perfect subject to learn more about finance, a field I had
 been interested for some time at that point, and get exposed to people that are
 knowledgeable, to cultivate my budding interest in the field.
 During my internship, I was part of the *Global Execution* team at IMC, a new
 team that was tasked with working on projects of relevance to multiple desks
 (each of whom is dedicated to working on a specific exchange). Their first
 contribution being to write the *Execution Gateway*, a piece of software that
 should act as the intermediary between IMC's trading algorithm and external
 exchanges. This is meant to replace the current pattern of each auto-trader
 connecting directly to the exchanges to place orders, migrating instead to a
 central piece of infrastructure to do the translation between internal IMC
 protocols and decision, and exchange-facing requests.
 As part of this migration process, my mission was to work on providing a simple
 framework that could be used to measure the performance of such a gateway. To do
 so, we must be able to instrument one under various scenarios meant to mirror
 real-life conditions, or exercise edge-cases.
 This lead me to first get acquainted with the components that go into running
 the gateway, and what is necessary on the client side to make use of it through
 the *Execution API*, which is in the interface exposed to downstream consumers
 of the gateway: the trading algorithms.\
 Similarly, I learned what the gateway itself needs to be able to communicate
 with an exchange, as I would also need to control that side in the framework to
 allow measurements of the time taken for an order from the client to the
 exchange.\
 To learn about those dependencies, I studied the existing tests for the
 *Execution API*, which bundle client, gateway, and exchange in a single process
 to test their correct behaviour.
 Once I got more familiar with each piece of the puzzle, I got to work on writing
 the framework itself. This required writing a few modules that would provide
 data needed as input by the gateway as a pre-requisite to being able to run
 anything. With those done, getting every component running properly was the next
 step. With that done, the last part was to generate some dummy load on the
 gateway, and collecting its performance measurements. Those could be analyzed
 offline, in a more exploratory manner.\
 With that work done, I had delivered my final product relating to my internship
 subject. Taken from a first *Proof-Of-Concept* to a working end result. Along
 the way I learned about the specific needs relating to the usage of those
 benchmarks, and integrated them in the greater IMC ecosystem.
 With the initial work on the benchmark framework done, my next area of focus was
 to add compatibility testing for the gateway. The point of this work was to add,
 as part of the *Continuous Integration* pipeline in use at IMC, tests that would
 exercise the actual gateway binaries used in production, to test for both
 forward and backward compatibility. This work fits perfectly as a continuation
 of what I did to write the benchmark framework: both need to run the gateway and
 generate specific scenarios to test its behaviour. This allowed me to reuse most
 of the code what I had written for the benchmark, and apply it to writing the
 tests.\
 The need for reliable tests meant that I had to do a lot of ground work to
 ensure that they were not flaky, this is probably the part that took longest in
 the process, with some deep investigations to understand some subtle bugs and
 behaviours that were exposed by the new tests I was attempting to integrate.
 Towards the end of my internship, I presented the work I did on the framework to
 other developers in the execution teams. This was in part to showcase the work
 being done by the *Global Execution* team, and to participate in the regular
 knowledge sharing that happens at IMC.
 Joining a company during the COVID period of quarantines, working-from-home, and
 the relatively low amount of face-to-face contact high-lighted the need for
 efficient ways of communicating with my colleagues. Being part of a productive,
 highly driven team has been a pleasure.
 Overall, this internship was a great reaffirmation of my career choices. This
 was my first experience with an actual project dealing with performance of
 something actually used in production, and not simply a one-off project to work
 on and leave aside. Being able to contribute something which will be used
 long-term, and have impactful consequences, reassures me in the choice of being
 a software engineer, and the impact of my studies at EPITA.
 \newpage
 # Thanks and acknowledgements
 First off, I would like to thank Jelle Wissink, an engineer from the Global