For physical heroes, I do them somewhat evenly since I skill each turn. For mages, just focus on their staffs (spirits to about level 20 or so, but no heavy investment).

If your physical heroes still do physical attacks (so without lb spirit and herja combo), you should work on swords more. Spirits are a little more expensive and take a little longer to level up.