From 568ba04ce74e0f47baa62a2273e2dbbd8778becd Mon Sep 17 00:00:00 2001 From: Bruno BELANYI Date: Thu, 22 Jul 2021 20:27:41 +0200 Subject: [PATCH] report: add engineering practices section --- report.md | 110 +++++++++++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 109 insertions(+), 1 deletion(-) diff --git a/report.md b/report.md index dc2f958..b78c309 100644 --- a/report.md +++ b/report.md @@ -323,7 +323,115 @@ FIXME # Engineering practices -Problematic: development of a benchmark framework +## Delivering a project from scratch to completion + +During the course of my internship, I have had to deliver a product from its +first *Proof of Concept* to a usable deliverable, going through various +iterations along the way. + +This process started with me getting familiar with the IMC code base, coming up +to speed with the tooling in use, some of the knowledge needed to work on the +existing code, etc... Once I was productive, I could start exploring the +necessary component needed to write a benchmark using the gateway binaries. This +initial POC was written by gutting some existing code in use for the Execution +API tests, splitting it into client and exchange code on one side, and gateway +code on the other. + +From this initial experiment, I got more familiar with the different components +that would go into an implementation of the benchmark. With the help of my +mentor, I could identify the needed dependencies that would to be provided to +the gateway binary in order to instrument it under different scenarios. + +I worked on writing those components in a way that was usable for the benchmark, +making sure that they were working an tested along the way. One such component +was writing a fake version of the RDS (FIXME: what does it mean again?) that +would be populated from the benchmark scenario, and provided the information +about financial instruments to the gateway in order to use them in the scenario, +e.g: ordering a stock. + +I went on to write a first version of the benchmark framework for a specific +gateway and a specific exchange: this served as the basis for further iteration +after receiving feedback about my design. Writing a second benchmark for a +second exchange and gateway lead to more re-design. + +The basic components of the benchmark framework were useful outside of their +original intended purposes, as I could reuse them to write the compatibility +tests, which were similar in the way they make use of the gateway to run a +scenario and verify the correct behaviour of our system. + +With all that work accomplished, I now needed to share my knowledge to make sure +that my mentor and I were not the only people who knew that this framework +existed, how to use, or how to work with it. To that end, I documented by +writing some useful tips I could give to help debug the benchmarks and tests +using gateway binaries. I also gave a presentation at the end of my internship +to demonstrate how to run a benchmark, and explain the main components of the +framework. + +I have delivered a complete, featureful product from scratch to finish, complete +with documentation and demonstration of its use. This is at the heart of our +schooling at EPITA: making us well-rounded engineers that can deliver their work +to completion. FIXME: wax a bit more poetics + +## Acquiring new skills and knowledge + +IMC is part of the financial tech sector, taking a position of market-maker on a +large amount of exchanges world-wide. + +The financial sector, even though I was attracted to it by my previous exposure +from conference sponsors and blog posts from engineers in the sector, was still +something I was not familiar with when first joining the company. + +There is a large amount of vocabulary and knowledge specific to this industry, +not even to mention the infrastructure and tooling in use at IMC, which while +not specific to this industry in particular, was still foreign to me at that +point. + +Before starting my internship, I was advised to read a book about high frequency +trading, which gave me some context on how exchanges work, and a few important +words that are part of the financial vocabulary. In addition, I learned about +IMC's trading infrastructure through a number of presentations that my team lead +organised with new hires during the beginning of my internship. This not only +gave me more context about what part of the existing infrastructure was aimed to +be replaced by the new *Execution Gateway* and the *Execution API*. I also got +to learn about some of the basics of pricing theory, which underpins our whole +strategy layer to come up with an appropriate valuation for any product we are +interested in trading. + +I got to further learn about trading and option theory during a training week +organized with a dozen other summer interns: we were taught some of the +mathematics that form the basics of valuation reducing risks in trading, the +associated vocabulary, and apply them during workshop exercises in trading with +the other interns. + +On the technical side, not only did I learn about the software stack in use at +IMC, as I worked on more and more parts of the code base I discovered new +tooling put in place to work and debug parts of our stack that are too costly to +setup or use on any dev computer. One such solution is the *fullsim* system, +which allows us to simulate our FPGA engines in software, to allow developers to +debug issues with FPGA interaction without having to fiddle with actual FPGA +cards or know how to use them. I also introduced my colleagues to new tools that +they were unaware of, the most prominent being the one I always reach for first +when trying to debug a piece of software: `rr`, which allows one to record a +program's execution and run it under a debugger in a totally deterministic +manner: it allows replaying and rewinding execution at will, making it a great +asset when dealing with issues that are sporadic, or require tricky timings like +networked systems. + +IMC encourages knowledge sharing across all teams, it permeates the company +culture, and shows in many ways. An execution engineer is encouraged to learn +about trading, which gives us more context when interacting with traders, +spotting mistakes in new strategies or guiding which features would make sense +to write next. Catch-up meetings are organized regularly between teams. +Presentations are given to teach people about the work that is being +accomplished to improve every part of our infrastructure, from deployment +tooling, developer productivity, to new strategies or components of our systems. + +Thanks to my well-rounded education, not only do I feel comfortable being +exposed to all this information. But I have felt confident that I fit in from +the start, and could keep pace with the information that was fed to me. I am +able to pick up those new skills fast, because EPITA taught us the most +important skill of this trade: I learnt how to learn, and how to flourish while +doing so. # Illustrated analysis of acquired skills