What level do the four castles become when turning to a free?
Do they go back to 40 or stay at whatever you got them to?
If the former then that is a lot of time and resource lost...if the latter then that provides a whole load of juicy targets for people.
If the latter I can see big alliances waiting and watching for fortresses to be made and swooping on the frees. They would have the military might to hold them as opposed to smaller ones