- Responsibilities: create Cloud-based systems to replace Goldman Sachs (GS) proprietary systems, develop tools for quants
- Developed a distributed compute system on AWS from the ground up, replacing a GS system.
- Considered off-the-shelf solutions using e.g. Slurm, AWS Batch/ECS. They did not meet stringent security and confidentiality requirements (e.g. authentication, PII) as well as latency (<100ms to start tasks) and throughput (peaks of 10k+ jobs submitted per second) requirements. As a result, we went for a semi-custom solution
- Integrated into existing UIs and took into account feedback from users (quants)
- Wrote scripts to pin down external package versions across multiple Conda environments, to ensure consistency
- Configured Cloud-based, ephemeral Linux containers as development environments (Gitpod/Ona with PyCharm).
- Delivered a “battery-included” experience for the main quants workflows, working with the infrastructure team to pre-install on the corporate fleet
- Regularly worked to improve CI pipelines success rate and build time (e.g. with smart, minimal dependencies)
- Engineering support for end of day risk calculation and business hours CI pipelines
✓ Python, Conda, Rust, AWS (EC2, MemoryDB, CloudWatch), SecDB, Gitpod/Ona, TeamCity