1. You will need a shovel, hand pruners and a pruning saw. A pruning saw does a much better job than a carpenter's saw, and isn't very expensive.
2. Cut off all the small branches with the pruners. Leave most of the trunk (or biggest branches) intact, up to three or four feet from the ground, if possible.
3. Dig a trench around the bush about a foot deep. Use the pruners to cut off big, long roots.
4. Using the intact wood for leverage, pry it out of the ground. If you don't weigh much, get a bigger person to do it.
5. Once the roots are exposed, you will probably need to use the pruning saw to cut through the biggest roots. Try to cut them as deep as possible to prevent regrowth from the roots.
6. Pile the dirt back over the hole. Hopefully, it will look like your ugly bush was never there!