Hints:
Firing directly into the stern or bow of opposing ships is called a rake. Raking fire is by far the most effective combat option in the game. Try for these types of strikes whenever possible. You can experiment with the Only Fire if Raking option to see when it can be used effectively. Be careful not to overuse it--you could take critical damage while waiting for the perfect broadside.
If you end up grappling an opponent's ship or get your masts fouled together, there will be a melee. Be sure to allocate a portion of your crew for boarding parties. While all crew are counted in the melee, those assigned to boarding parties will be twice as effective as the rest of the crew.