The term frag originates from military slang, where it is short for fragmentation. In Doom and other FPS games, frag refers to the act of killing another player's character in the game. In cooperative mode, there is a frag count indicating how many times you killed your allies, accidentally or otherwise. When deathmatch mode was implemented, frags were used as a measure of score. Each kill of an opponent earns a frag, while suiciding subtracts one. When the frags are displayed at the end of the deathmatch, whoever has the most frags is the winner.
Frag is occasionally extended to mean killing monsters as well.