KCG - Knight Capital

[Jurgis]

Possibly not very relevant to investing.

 

There were two causes for the debacle:

1. Data structure (flag) reuse for different functionality in new code. The old stuff that was not going to be used in new code was re-purposed for different functionality in new code. This is usually bad engineering as we can see from the results.

2. New code was supposed to be deployed on all machines. Instead it was deployed on 7 out of 8. Old code was still on the 8th machine. This is human error in deployment.

 

In conjunction this blew up, since the 8th machine was interpreting the new stuff the old way, which caused the order deluge. People could not immediately figure out what was happening and made some bad decisions (i.e. roll back new machines to old code) that exacerbated the problem. These decisions cannot be blamed: debugging while your system is losing millions $ per minute is not easy. The right decision was to pull the plug and debug offline, but this looks easier after the fact than in the moment.

 

Item 1. can be improved via better software engineering practices. Hopefully they adopted them.

Item 2. can be improved via better deployment practices. Hopefully they adopted them.

There are no guarantees, but they probably won't experience similar issue again.

 

The above is simplified summary. For more depth, there are blog posts and post mortem analyses online (my understanding is based on https://dougseven.com/2014/04/17/knightmare-a-devops-cautionary-tale/ among others; there's also official SEC investigation document that is not very readable). Some people made erroneous posts after debacle that were not corrected when facts emerged (zero hedge, for example).

[marazul]

Thank you for the post Jurgis, adds clarity.