Read on Twitter
1 /
the one where they lost $2 million in 2 days (and made it back in 2 weeks)
the one where they lost $2 million in 2 days (and made it back in 2 weeks)
2 /
by far, the biggest existential threat to an automated trading system is algorithmic logic error
trading performance matters, but that's the easy part
a single overlooked logic bug could sink you in seconds
by far, the biggest existential threat to an automated trading system is algorithmic logic error
trading performance matters, but that's the easy part
a single overlooked logic bug could sink you in seconds
3 /
history illuminates
i) knight capital lost $440 million over 4 million executions buying high, selling low
https://bit.ly/3i7Bg4Z
ii) a hk market maker recursively added 1-lots to bid in kospi futures >100k times, filled for >US$3bn notional https://bit.ly/3i1XHsp
history illuminates
i) knight capital lost $440 million over 4 million executions buying high, selling low
https://bit.ly/3i7Bg4Z
ii) a hk market maker recursively added 1-lots to bid in kospi futures >100k times, filled for >US$3bn notional https://bit.ly/3i1XHsp
4 /
for automated systems, every time we add a new model param / feature:
i) what could go wrong here
ii) can we catch this without obstructing the hot path?
iii) thorough regression testing
for automated systems, every time we add a new model param / feature:
i) what could go wrong here
ii) can we catch this without obstructing the hot path?
iii) thorough regression testing
5 /
take for example a typical crypto api http response:
i) what if response is str instead of expected float
ii) what if json format is surprisingly reshuffled
iii) what if new unseen fields are present
iv) what if exchange sends an ice-cream sandwich
and so on and so forth
take for example a typical crypto api http response:
i) what if response is str instead of expected float
ii) what if json format is surprisingly reshuffled
iii) what if new unseen fields are present
iv) what if exchange sends an ice-cream sandwich
and so on and so forth
6 /
try as we might, it's _difficult_ to pre-empt every potential bug in a distributed system
and for @mgnr_io, this manifested on march 12 / 13 (crypto black thursday / friday)
try as we might, it's _difficult_ to pre-empt every potential bug in a distributed system
and for @mgnr_io, this manifested on march 12 / 13 (crypto black thursday / friday)
7 /
as $BTC and $ETH took a violent swan dive, exchange matching engines all but gave up
@BitMEX pulled the plug. @KrakenFutures liquidity vanished - books were thousands $ wide
across the gamut, in the order of 10s of unseen and undocumented errors were coming down the pipes
as $BTC and $ETH took a violent swan dive, exchange matching engines all but gave up
@BitMEX pulled the plug. @KrakenFutures liquidity vanished - books were thousands $ wide
across the gamut, in the order of 10s of unseen and undocumented errors were coming down the pipes
8 /
staying afloat, the first $1k downtick yielded extremely profitable trading conditions, and at that point we were +60 BTC
but then the violence accelerated
staying afloat, the first $1k downtick yielded extremely profitable trading conditions, and at that point we were +60 BTC
but then the violence accelerated
9 /
we frantically rebalanced collateral across hedged long-short accounts (eg long on @Bybit_Official, short on @FTX_Official)
we were flat delta (no net directional exposure)
but blockchains were bottlenecked, and we could not collateralize fast enough
we frantically rebalanced collateral across hedged long-short accounts (eg long on @Bybit_Official, short on @FTX_Official)
we were flat delta (no net directional exposure)
but blockchains were bottlenecked, and we could not collateralize fast enough
10 /
adding fuel to the fire, exchanges failed to update positions, failed to publish positions, published stale positions
when your system has an inaccurate picture of the state of the world, it's about to get tricksy...
adding fuel to the fire, exchanges failed to update positions, failed to publish positions, published stale positions
when your system has an inaccurate picture of the state of the world, it's about to get tricksy...
11 /
BAM! liquidated on @BitMEX
BAM! liquidated on @OKEx
bam, bam, bam...
cascading fun for the whole family
BAM! liquidated on @BitMEX
BAM! liquidated on @OKEx
bam, bam, bam...
cascading fun for the whole family
12 /
meanwhile elsewhere, we were liquidation beneficiaries
the circle of life
meanwhile elsewhere, we were liquidation beneficiaries
the circle of life
13 /
after the carnage was over, we took stock
down $2 million
define flesh wound?
we were lightly toasted, but still standing. we'd been there before, and probabilistically, we'd be there again!
so we locked our sights on working out of the hole
after the carnage was over, we took stock
down $2 million
define flesh wound?
we were lightly toasted, but still standing. we'd been there before, and probabilistically, we'd be there again!
so we locked our sights on working out of the hole
14 /
the optimist wins:
i) it was clear that a number of other competitors were catastrophically wounded (completely out of market). books were _thin_
ii) markets were jumpy and pay-thrus frequent. with a strong model this should be ideal conditions for market-making
the optimist wins:
i) it was clear that a number of other competitors were catastrophically wounded (completely out of market). books were _thin_
ii) markets were jumpy and pay-thrus frequent. with a strong model this should be ideal conditions for market-making
15 /
we de-risked our quote qty to 1/5th the previous day, adjusted some model parameters, and got back in market
immediate results
the next 2 weeks were like a tour of the tropicana factory. we manufactured pure, freshly squeezed, 100% JUICE
gulped down 2 bucks, then some
we de-risked our quote qty to 1/5th the previous day, adjusted some model parameters, and got back in market
immediate results
the next 2 weeks were like a tour of the tropicana factory. we manufactured pure, freshly squeezed, 100% JUICE
gulped down 2 bucks, then some
16 /
lessons learned:
i) re-doubled efforts to handle previously unseen errors. cannot overstate importance of graceful error handling in distributed systems
ii) increased per-venue collateral buffer to 3x previous. tradeoff lower ROE, but liquidation is literally a hard stop
lessons learned:
i) re-doubled efforts to handle previously unseen errors. cannot overstate importance of graceful error handling in distributed systems
ii) increased per-venue collateral buffer to 3x previous. tradeoff lower ROE, but liquidation is literally a hard stop
17 /
thanks for reading! follow along for more reflections and stories from crypto HFT / algorithmic trading
thanks for reading! follow along for more reflections and stories from crypto HFT / algorithmic trading
