Forts are awesome. At the beginning it's easy to manage multiple castles but once you reach over 50 I suggest converting them into fort if you can. More troops available, faster rss production and more storage pays off.
Cost is nothing at that point and benefits definitely justify time and silver spent on forts.