Yes, Legion of Shadow can be quite punishing (particularly in Act 2) if you don’t keep on top of your speed attribute. As other players have stated, the priority order for items should really be speed first, then your main damage attribute, followed by armour. (In HoF, this is less rigid as the itemisation is much improved!). It sounds like you might have chosen armour items (with low speed) over items that might have boosted your speed. You don’t want to fall behind, so always keep an eye on the speed (or level, if you wish) of the monsters in your current quest. If they start creeping ahead, then start looking for that next speed item and equip it asap.
A rough guide for Act 2:
Green quest – aim for 6-7 speed
Orange quests – boost to 8-9 speed
Blue quest – 9-10 speed
Red quest – boost to 10-11 speed
Remember that the abilities you choose can also play a part in your choices, as well as your career. Also worth remembering that the legendary monsters and the arena side quest are optional and quite difficult challenges. Not all heroes/builds will be able to complete those.
Hope that helps. I’m sure other players might chime in with further advice.