Skill attacks actually uses both the skill attack and physical attack stats. The skill attack stat of course has a lot more impact on the damage dealt by a skill attack than the physical attack stat.