The following thread is going to be a barebones basic primer - on Systematic Trading from research to live deployment. Will do this in parts if the no. of tweets exceed beyond refreshable limit. :) Seasoned pros here, please pitch in if you see some scope for improving the thread

First of all, I am going to write this thread in the assumption that it will be 100% mechanical trading with no human intervention. Bringing discretion into mechanical trading is not something I am yet familiar with, so, for the purposes of this thread it will be 100% mechanical.

So, before creating a system, there are majorly two possible classes of systems.
1. Trend Following - where you follow the trend and take trades that hopefully align with the trend and ride it.
2. Mean Reversion - where you take trades betting on price reverting to mean.

1. When it comes to Trend Following, you want to identify the market's current trend, and ride it. You can't catch top and bottom, so you'd want the meat of the middle chunk of the move.

2. When it comes to Mean Reversion, the same applies. Just that you'll be trading against the trend. A lot of people say trend is your friend/don't trade against trend, but if you do it right, you can trade against trend and make money, that's where mean reversion trading happens

3. So, before talking about research & development of trading systems, I would like to put a word about technical analysis, indicators, patterns, etc. Systematic trading is applicable to any system that can be codified objectively, in a fixed manner. No two ways about it.

4. Classic technical analysis - I don't know whether it works or not - patterns like cup and saucer, trendlines, flagpole, pennant and all that. These are all highly subjective and more of an art than based on science/math. So, I ll just keep these artistic things aside for now.

5. Price, Volume, Volatility. These three things are more than enough to create an interesting, profitable system. The higher you go in the order of derivatives of these three factors, the lower your returns, the more lagging the system.

6. So, the closer the system is to the three main factors for developing a system, the better. Essentially the following comprises a proper system.
1. Rules for entry (15-20% importance)
2. Rules for exit (40-45% importance)
3. Position sizing rules (40% importance)

7. The entry is (in my opinion, and opinions can differ) the least important part of a trading system. It's very important, i am not differing at that. But compared to the other two - it has the least weightage of importance. This will be discussed separately in a thread later.

8. The exit rules - imo is the most impt part of a system because it is where you manage your risk along with position sizing rules. A successful trader is predominantly a successful risk manager - and you manage risk by deciding where to keep SL/Target.

9. Usually trend following systems advocate TRAILING profits through the Trail Stop Loss mechanism. You don't set a hard target as a trend can extend to god knows what area and if you set a hard target, you'll most likely kill your system before you even begin.

10. On the contrary, the best mean reversion systems I have seen have fixed hard targets (and some of the famous Mean Reversion system traders I know don't keep a super-hard stop loss. Not suggested, but just an FYI).

11. Trend following systems work in trending market, don't work much in rangebound market. Mean reversion systems for most part work across the market cycle, but performance varies based on volatility. This is very important to know when you start creating your systems.

13. Position Sizing rules basically tell you how much of your total portfolio/account you'll be using for each trade, and this combined with Exit rules form the risk management part, decides the efficiency and robustness of your system.

14. With super-tight position sizing rules and risk management, you could compound your account multi-fold over the years with the right system. Now, let's see how to go about research & development of any system.

15. You don't need to ask any of the top traders what their system is, how they trade, what their capital is, etc. All that is useless to you. Even if they hand their system to you - you wouldn't be able to trade it successfully without putting in all the work yourself.

16. So, disclaimer: Don't expect to find an idea and test it for it to turn out to be the most successful system from the get go itself. I tested 60+ systems (available online posted as profitable - mostly curve fitted) before I found one that was decently profitable.

17. I love the way @bhatiamanu had put it in one of his QNAs in reddit. You have a hypothesis, you test the hypothesis, and if there's some validity/correlation, you test further and identify ways to trade the hypothesis.

18. So, where do you find ideas to test? You can start with traderji forum, zerodha interviews of 60 day challenge winners, forexfactory, and such trading forums where people have posted NUMEROUS good ideas. Takes a little bit of tweaking, but you'll get there.

19. For the sake of this thread, I'll pick one of the simple systems for trading as given by @AlBrooksTrading.

Timeframe - 5min
System basis: In the first 90 mins of the market trading, the high or low of the day has already been made (90% probability).

20. So, if the system has made the high/low of the day by first 90 mins, breakout in any direction is likely to continue in that direction or be range bound, but won't break the lowest registered in the first 90 mins of trading day. AlBrooks says this is with 90% probability.

21. So, what would be the rules for the system?
Entry rule: Buy on successful close above first 90m high/Short on successful close below first 90m low.
Stop Loss: Slightly below the first 90m low/above the first 90m high respectively.

22. Exit: At the end of the day (around 3.15pm) or SL hit.
You could tweak this to trail SL, but there will be whipsaws and you might lose a chunk of the move in highly volatile market.

23. If you trail SL, you could test accordingly. Depending on the capital you have, you could start with 1 lot futures, and add further lots as you add profits in the system. This is according to each system. Now,you have the rules - entry/exit and SL. Next what? Backtest Prelims

24. As @madan_kumar suggested, when you have an idea - test it for 1 quarter per year for last 10 years. There are different ways to go about backtesting, each method has its pros and cons. So for this idea, take the NIFTY/Bank Nifty IEOD data for last 10 years and test.

25. Two ways to test.
First, MANUAL bar by bar testing. Time consuming. Takes atleast 2-4 hours with full focus to test 1 quarter worth of data. So if you spend 2 fast hours per day, you could be done in 10 days. Go through price chart, bar by bar, and test.

26. Second method - programming. Amibroker (uses afl language similar to C++) or Python (c++/Java works too) or Excel (best to start with). You code the rules of the system, import the price data, run the system through the data for those quarters, and get the results.

26. Let me cover the first method first. Manual bar-by-bar testing. If you have official amibroker version, you have bar-by-bar replay of historical data just like how it would have played in real time. That's even better as you see min-by-min price movement.

27. It looks like this. Set the from and to date. Set a comfortable speed and start - and pause as and when you mark the trades taken. Keep an excel sheet ready to mark entry/exits.

28. Do this for atleast 5 random years in last 10 years to check how the system has performed. If the results are promising, go further. Select random quarters for each year. Put a spreadsheet of entries/exits to make further calculations later. It would look like this.

29. Once you are done with the step mentioned above, if the system shows promise for most part of the 5 quarters, extend testing to another 5 years - 1 random quarter per year. If all the 10 quarters show promise (net profitable) - test manually each year for the last 10 yrs.

30. Once you have all the results in your spreadsheet - up until total profit points, cumulative points made (sum total result of all trades for a year), after you have done the manual backtest for all 10 years, you then look at the different parameters that are important.

31. This is amibroker backtest report. To begin with, you should look at your total no of trades, the win rate, the net profit points after accounting for commissions and slippage, win ratio, average points per winning trade, and similar data about losing trades.

32. You could use the relevant formulae on excel for all these parameters you want to look at, and find out the results. Something like this, but a bit more elaborate.

33. It's preferable to have a system with a win rate of at least 35-40%. Any less, you're going to have to manage huge drawdowns. The lesser the win rate, if such a system is profitable, it means very few trades are responsible for most of the profits.

34. That also means you have to show up and take every single trade throughout the year. Not taking one trade could mean the difference between an incredibly profitable system, and an average FD beating system.

35. Like if you have a trend following system that made an average of 3000 points in 2019, 700 of that would have come from the corporate tax cut day, and if you'd missed that day due to flu or something else, your system's risk adjusted returns will be very different.

36. As you look at win rate, you should also look at max drawdown, average winner (in pts) and average loser, the different ratios between winners and losers, etc., to understand your system's edge. But to begin with, start with systems that have atleast 40% win-rate.

37. That said, win-rate alone is not important, but in tandem with the other parameters I have mentioned above, could decide the worthiness of a system. It helps to have a thorough understanding of these different parameters and performance through the 10 year period.

38. This is like an overview of the process. Some people split 10 years period as 5-6 years of in-sample data (where they optimize/fine-tune the system parameters, add conditions, etc.,) and test the system in remaining 4-5 years data.

39. The simpler the rules, the better the system. With each single parameter, additional rule or condition you add into the system, the complexity increases exponentially. This would lead to curve-fitting, over-optimisation, and failure in real-time trading most possibly.

40. Once you have done all these, and you're happy and satisfied with the system, confident about its validity, etc., you start the process of live trading. I would suggest taking at least 50 trades with 100% precision of entry/exit without disrespecting the system with 1 lot.

41. The reason I suggest that is because we should test the validity of the system in real-time, no paper trading - actual live trading gives us the best transparent look into the system. Taking 50 trades ensures that your results aren't due to chance occurrences.

42. Once you have achieved good results in live-trading with just 1-lot (futures or whatever instrument you have tested your strategy in - that you trade), you then start ploughing back the profits into the system, and scale up.

43. Before scaling up, revisit the excel sheet for all 10 years, and test the system's profitability for different leverage %, max drawdown %, etc., to understand how much leverage is good for you with the specific system. You may think a drawdown of 20% is okay, but it won't be.

44. A drawdown of 50% requires 100% profit to be made to get back to the same level. You may think 20% drawdown is okay, but most people can't even stomach a drawdown of 5-7% in their portfolio. The emotional impact of percentage based actual INR amount gets worse as you scale up

45. You may think I am okay with 20% dd, and put up 10k rupees, losing 2k may not be much. But losing 2 crore out of 10 crore would hurt a lot. So, think about the system from an absolute capital perspective and it doesn't hurt to start off from a conservative standpoint.

46. So, understand that the lesser the historical max DD, the better. Also, leverage is to be treated with absolute respect and carefulness. One mistake can cost your year's worth of profits. Which is why you need to test different outcomes of your system with different lever %.

For instance, on a capital of 10 Lakh, if i traded 1 lot on this particular system i tested, it had a max drawdown of 2.4%, but on 8 lots, it was about 12%. Can I stomach 12%? I don't think so. 7.5% ? I'd be comfortable with that. So I'd choose 4 lots.

47. No matter what the historical drawdowns are, your future dd can be better or worse but won't be statistically far off. So, if i take up 4 lots of Nifty Futures, at 10L capital, I may end up with 7.5% +/- 3% give or take. I'm fine with that coz i know how much i can stomach.

48. Likewise, know your risk tolerance/appetite, and decide leverage accordingly. Never hurts to be conservative initially and slowly ramping up as you gain more confidence. Always err on the side of caution. Your risk adjusted returns are far more important than absolute returns

49. Once you have gone through these first 50 trades, and you have decided how much leverage you'd use based on historical performance and drawdowns, you start trading the system. It's pertinent to be absolutely ruthless with execution. No place for emotion. No interference.

50. If you have done everything right up until this point, you then monitor the system and watch your account grow, strive to execute the trades flawlessly as and when the system throws you a trade. Scale up as per your system and capital.

51. For ex: In one of the systems I tested, I decided on a leverage of roughly 50% (trading 2 lots Nifty Futures with 10L capital account) and for every 5 lakh profit made, I'd add one more lot in size. Something like this, very conservative, but the system worked beautifully.

52. Like this, decide on a compounding plan. You can be conservative or aggressive - depends on your capital allocation and risk appetite. Once this is in place, you compound your capital accordingly. Alongside, you monitor the system for relevance and when it needs deprecation.

53. You can also spend your time researching and developing other systems (non-correlated) to trade and have a portfolio of strategies trading at any time (depending on your capital). As and when a system's edge disappears, retire the system, and plug in another relevant system.

54. If you know how to work with Excel, Python/C++/Java or Amibroker, you could test a strategy all within a day or two. You could go through extensive backtests. But execution still needs to be done manually unless you have a way to completely automate the system.

55. But if you were to manually execute the signals, skipping manual back testing will make for lesser confidence in the system when push gets to stove. So, I'd always advocate for manual backtesting and then live trading.