There's 2 main ways to go about selling gems. You either go for the gems highest in demand, or you cater to a more specific customer base.
In simpler terms-- You can sell Runed scarlet/cardinal rubies which every single caster (including healers to some extent) uses. The demand for these is higher, but so is the supply because the majority of people follow this same reasoning. What this means is that you'll rapidly sell 100% of the gems made, but you'll often end up involved in an undercutting war.
The other way to go about it is to go for very specific gems. This means cutting gems that only one (or a few) specs use, either during early stages of gearing or near BiS levels (these are the 2 stages of gearing where you regem the most, especially the latter). The demand isn't as high for these, therefore the supply is lower too. You can get better profits per gem this way, selling them at a higher price.
You may have noticed this trend already, because generally tank consumables/enchants/gems tend to cost a bit more than dps ones. Reason for that is there's a considerably larger number of dps characters (and healers for that matter) compared to tanks.
I haven't been very specific as to which gems to sell exactly, but generally speaking:
The standard pure-stat gems will often be highest in supply so you might not always get the highest profit out of them. These include: Spellpower/Strength/Armor Penetration/Agility/Stamina/Resilience/Haste/Intellect.
Specific pure-stat gems that might yield higher profit could be: Expertise/Hit/Spell Penetration/Attack Power
The standard hybrid gems might also follow that same trend. These include: Sp+crit / Sp+haste / Sp+spirit / Sp+resilience/ Str+crit / Str+haste.
Specific hybrid gems that might yield higher profit could be: spell pen+haste/sp/hit / agi+hit / exp+hit / str+hit
Try to study the PvE and PvP guides a bit and see which gems every class and spec uses, and you can sort of gauge the supply/demand for each. You can then confront the AH and decide what to sell.