I often hear this notion of "To fix performance we will need to make heroic changes!"(re-write/design/architect), often you can make a big difference with profiling and fixing stupid problems. Examples from current work something:
1/N
1. Break big query into small ones: result was too big and taking forever to get back (stupid upstream)
2. Run those in parallel(because now pipeline is drip fed by small queries)
3. Cleanup empty tmp directories(they still fill up your hard drive)

2/N
4. Use cheap metrics(tracking disk space recursively is expensive), also keep an eye on metrics cost
5. Compile and provide lib X with a native lib so it's faster.
6. Re-balance thread pool sizes.

None of this is major, but the effects are huge.
3/N
Point is, a little love goes a long way. Measure, profile, tweak, iterate... There's lots of wins which require very little effort. No assembly required :P
4/4
You can follow @nitsanw.
Tip: mention @twtextapp on a Twitter thread with the keyword “unroll” to get a link to it.

Latest Threads Unrolled: