The correct way of saying May Allaah reward you
Jazaak Allaah khayr literally means: May Allaah reward you to a male.
Realistically, jazaak should be pronounced as jazaaka but the a is usually omitted when speaking to a male (this is the case with many spoken Arabic words), but left when written. On the other hand if jazaak were to be written in Arabic, it would be unclear who the writer is referring to, is it a male or a female? Also, Allaah should be said as Allaahu. This is due to grammatical reasons, where the u displays the doer (the one who will be rewarding), i.e. Allaah (swt).
Jazaak comes from the root word jazaa which according to the popular Arabic-English dictionary, Al-Mawrid, has two meanings that are completely opposite to eachother!
Thus jazaa can either mean reward/requital OR punishment/sanction. For an even more accurate reference, one just has to turn back to the Quraan and see how Allaah (swt) uses the word jazaa regarding the reward for the believers and the punishment for the disbelievers in the Day of Judgment, may Allaah (swt) save us from that, aameen.
Jazaak literally means May he reward in a requesting sense. Thus the person saying that is making duaa. Obviously duaa can only be made to Allaah. That is why Allaah is immediately mentioned after jazaak, for He is the only one who can reward with good or bad.
If you were to say to someone: Jazaak Allaah, the person would, in almost all cases, think that you are asking Allaah to reward him/her with good. This is the case with an Arab as well. But if you were to pay attention closely to what you are saying, you will see that you are not emphasizing what you would want Allaah to reward that person with. Imagine a person that is very angry at someone, telling him: Jazaak Allah sharran! (May Allaah reward you with bad!). Although that is a very unlikely situation in shaa Allaah, it still shows us the importance of always being clear with our words and speech with other people, in order to prevent any misunderstandings.
Thus, what follows is the correct way of saying May Allaah reward you with good in different grammatical forms, i.e. to a man, woman, group etc.:
To a male: Jazaak Allaahu khayran
To a female: Jazaaki-llaahu khayran
To two people (whether male/female): Jazaakuma Allaahu khayran
To three men or more: Jazaakum Allaahu khayran
To three women or more: Jazaakumunna-llaahu khayran
In the case of a group of people, with only one man but many women, one must use Jazaakum Allaahu khayran (i.e. that which is used to three or more men). That is just how the Arabic grammar is.
When speaking about people that are not present, but invocating Allaah (swt) to reward them with good, the following is the correct way to make duaa for them:
To one man: Jazaahu-llaahu khayran
To one woman: Jazaaha-llaahu khayran
To two people: Jazaahuma Allaahu khayran
To three men or more: Jazaahum Allaahu khayran
To three women or more: Jazaahunna Allaahu khayran
In the case of a group of people, that has only one man but many women, one must use Jazaahum Allaahu khayran (i.e. that which is used to three or more men).
You may also like to add more emphasis to the goodness that you want Allaah swt to reward the person with. For example, after khayran you can say katheeran (plenty/abundant/many).
Alternatively, you could say after Allaahu khayrul-jazaa, which literally means the best of rewarding; thus when said altogether: May Allaah reward you with the best (type) of rewarding.
The Sunnah way to respond to someone saying Jazaak Allaah, is to say: Jazaana wa iyaakum (may Allaah reward us and you). For a shorter version, one can say wa iyaakum (and you).
This concludes the topic at hand in shaa Allaah. Whatever good that has been written is from Allaah. And whatever bad or mistake has occurred therein, is from me and Shaytaan.
Assalaamu alaykum wa rahmatullaahi wa barakaatuhu.