diff --git a/report.md b/report.md index 81aa418..0b7fadc 100644 --- a/report.md +++ b/report.md @@ -1,5 +1,95 @@ # Exec Sum +When making my decision to major in Image Processing and Image Synthesis, my +main motivator was my growing curiosity for high performance programming, as I +thought that some of the subjects being taught would lend themselves well to +furthering my understanding of this domain.\ +On my own, I discovered through a number of conference talks that I watched that +the field of finance provides many interesting challenges that aligned with +those interests. + +When the opportunity arose to work for IMC, a leading firm in the world of +*market-making*, I jumped at the chance to apply to them for my internship. When +asked for the kind of work I wanted to do during that time, I highlighted my +interests in performance, which lead to the subject of writing a benchmark +framework for their new exchange connectivity layer, currently in the process of +being created and deployed.\ +This felt like the perfect subject to learn more about finance, a field I had +been interested for some time at that point, and get exposed to people that are +knowledgeable, to cultivate my budding interest in the field. + +During my internship, I was part of the *Global Execution* team at IMC, a new +team that was tasked with working on projects of relevance to multiple desks +(each of whom is dedicated to working on a specific exchange). Their first +contribution being to write the *Execution Gateway*, a piece of software that +should act as the intermediary between IMC's trading algorithm and external +exchanges. This is meant to replace the current pattern of each auto-trader +connecting directly to the exchanges to place orders, migrating instead to a +central piece of infrastructure to do the translation between internal IMC +protocols and decision, and exchange-facing requests. + +As part of this migration process, my mission was to work on providing a simple +framework that could be used to measure the performance of such a gateway. To do +so, we must be able to instrument one under various scenarios meant to mirror +real-life conditions, or exercise edge-cases. + +This lead me to first get acquainted with the components that go into running +the gateway, and what is necessary on the client side to make use of it through +the *Execution API*, which is in the interface exposed to downstream consumers +of the gateway: the trading algorithms.\ +Similarly, I learned what the gateway itself needs to be able to communicate +with an exchange, as I would also need to control that side in the framework to +allow measurements of the time taken for an order from the client to the +exchange.\ +To learn about those dependencies, I studied the existing tests for the +*Execution API*, which bundle client, gateway, and exchange in a single process +to test their correct behaviour. + +Once I got more familiar with each piece of the puzzle, I got to work on writing +the framework itself. This required writing a few modules that would provide +data needed as input by the gateway as a pre-requisite to being able to run +anything. With those done, getting every component running properly was the next +step. With that done, the last part was to generate some dummy load on the +gateway, and collecting its performance measurements. Those could be analyzed +offline, in a more exploratory manner.\ +With that work done, I had delivered my final product relating to my internship +subject. Taken from a first *Proof-Of-Concept* to a working end result. Along +the way I learned about the specific needs relating to the usage of those +benchmarks, and integrated them in the greater IMC ecosystem. + +With the initial work on the benchmark framework done, my next area of focus was +to add compatibility testing for the gateway. The point of this work was to add, +as part of the *Continuous Integration* pipeline in use at IMC, tests that would +exercise the actual gateway binaries used in production, to test for both +forward and backward compatibility. This work fits perfectly as a continuation +of what I did to write the benchmark framework: both need to run the gateway and +generate specific scenarios to test its behaviour. This allowed me to reuse most +of the code what I had written for the benchmark, and apply it to writing the +tests.\ +The need for reliable tests meant that I had to do a lot of ground work to +ensure that they were not flaky, this is probably the part that took longest in +the process, with some deep investigations to understand some subtle bugs and +behaviours that were exposed by the new tests I was attempting to integrate. + +Towards the end of my internship, I presented the work I did on the framework to +other developers in the execution teams. This was in part to showcase the work +being done by the *Global Execution* team, and to participate in the regular +knowledge sharing that happens at IMC. + +Joining a company during the COVID period of quarantines, working-from-home, and +the relatively low amount of face-to-face contact high-lighted the need for +efficient ways of communicating with my colleagues. Being part of a productive, +highly driven team has been a pleasure. + +Overall, this internship was a great reaffirmation of my career choices. This +was my first experience with an actual project dealing with performance of +something actually used in production, and not simply a one-off project to work +on and leave aside. Being able to contribute something which will be used +long-term, and have impactful consequences, reassures me in the choice of being +a software engineer, and the impact of my studies at EPITA. + +\newpage + # Thanks and acknowledgements First off, I would like to thank Jelle Wissink, an engineer from the Global