Yeah, that part sucks. There tends to be some sort of pattern to who attacks you and when. I haven't found it out, but I have found that delaying building things as long as possible helps. Once you have a beast lair, those beasts go off and attack things. Maybe those things then retaliate and attack you.