These have been put on to most Area of Effect spells as of patch 2.2.
Each rank of each spell has a damage cap. Once this cap will be reached the spell does less damage to compensate so that it will not exceed its cap. The caps are not affected by gear or talents. So if you have good gear, you will reach the cap on (many) fewer mobs.
Apparently if mobs resist, the other mobs get more damage for that tick up to the cap. Also, crits seem to be able to exceed the cap for the mobs which are critted. (or maybe there is a crit cap also which just hasn't been reached)
Let's make a fictional AoE spell with a stated damage of 200 and an (unstated) cap of 2000.
- 1 mob = 200
- 2 mobs = 400
- 10 mobs = 2000
- 11 mobs = 2000
- 181.8 each.
Now add +1000 damage gear at 40% effect to damage (+400).
- 1 mob = 600
- 2 mobs = 1200
- 3 mobs = 1800
- 4 mobs = 2000
- 500 each.
- 11 mobs = 2000
- 181.8 each just like before
There is a nice list of empirical testing done on a forum post which I will try to find again. 
Blizzard: Rank 6 (1272 Frost Damage over 8 seconds) and 363 Bonus Spell Damage:
1-11 mobs, 236-237 damage per damage tick
12 mobs, 231-232 damage per damage tick
Arcane Explosion: Rank 10
Howling Blast(Rank 4) & Blood Boil(Rank 4)
3.3 will significantly alter the manner in which AoE damage cps are calculated. Instead of having a hard cap on the damage done, the maximum damage will be equal to the damage that would be dealt to 10 mobs hit by the ability. For example, if a spell does 1000 damage per target, it would deal 10000 total damage against 10 targets, and that would the maximum damage it could do against any number of targets (before critical effects are taken into account). Against 20 targets, the spell would do 500 damage per target, while against 40 targets the spell would do 250 damage per target. Against 10 or fewer targets, the spell would continue to do 1000 damage per target. Thus unlike pre-3.3 mechanics, the damage cap now scales with gear, the maximum number of targets that can receive full damage no longer scales inversely with gear, and the damage done once beyond target cap now also scales with gear.