ROTJ:
Luke was told all his friends were going to be slaughtered, and even after that, he was only trying to kill Palpatine, actively resisting a full-on battle with his father (space Hitler) whom he still believes in, and only breaks down once his sister was threatened. (1\\5) https://twitter.com/birdsith/status/1290870529310810112

TLJ:
A much older Luke who has long since learned the lessons from the above conflict senses his nephew (who has done nothing wrong at this point and isn't a proven threat) is having bad thoughts and immediately leaps to contemplating his murder without even talking to him. (2\\5)

The main difference is that Luke was much younger and under a much greater strain in ROTJ vs TLJ. It was the ultimate test of character for him. He was pushed very close to the edge, but he narrowly avoided going down that dark path and managed to redeem his father in the process

Meanwhile, Luke was in his home during TLJ with no immediate threat to his life or loved ones. Ben was deeply troubled, sure, but he had already redeemed his father, who was fucking Space Hitler, so Ben's bad dreams shouldn't really be enough to spook him.

This is his nephew. His sister and best friend's son. He should be trying to redeem that boy beyond the point of reason.

His downfall should have been not knowing when to quit and cutting his losses rather than completely going backwards on everything he learned in ROTJ.