Is there a finite group such that, if you pick one element from each conjugacy class, these don't necessarily generate the entire group?

No, this is impossible. This is a standard lemma, but I'm finding it easier to give a proof than a reference: Let $G$ be your finite group. Suppose that $H$ were a proper subgroup, intersecting every conjugacy class of $G$. Then $G = \bigcup_{g \in G} g H g^{1}$. If $g_1$ and $g_2$ are in the same coset of $G/H$, then $g_1 H g_1^{1} = g_2 H g_2^{1}$, so we can rewrite this union as $\bigcup_{g \in G/H} g H g^{1}$. There are $G/H$ sets in this union, each of which has $H$ elements. So the only way they can cover $G$ is if they are disjoint. But they all contain the identity, a contradiction. UPDATE: I found a reference. According to Serre, this result goes back to Jordan, in the 1870's. 


It is impossible. As I mentioned in the comment to Richard Stanley's answer, you are looking for a finite group $G$ with a maximal subgroup $M$ such that $M$ intersects every conjugacy class. Then $G=\cup M^g$ is the union of $M$ and its conjugates, which is wellknown to never happen. Steve 


The impossibility also follows from Jordan's lemma: Let $G$ act transitively on a set $\Omega$ with $\Omega:=n\geq 2$ then there exists a $g\in G$ such that $\chi(g)=0$ (here $\chi(g)$ denotes the permutation character). Here $\chi(g)$ denotes the permutation character. In fact with some additional work one can show that the proportion of elements $g\in G$ such that $\chi(g)=0$ is larger than or equal to $\frac{1}{n}$. So now let us see how Jordan's lemma implies that the answer is negative. So let $H$ be the group generated by the representatives of each conjugacy class of $G$ and assume that $H$ is a proper subgroup of $G$. Then we may look at the left action of $G$ on $G/H$. Since $G/H\geq 2$ and the action is transitive there exists a $x\in G$ such that $x g_i H\neq g_i H$ for each left coset $g_i H$. In other words for each $g_i$ one has that $g_i^{1}x g_i\notin H$ which in turn implies that for all $g\in G$ one has that $g^{1}xg\notin H$. Therefore the conjugacy class of $x$ does not intersect $H$ which is absurd. Note also that one gets the following corollary from the previous argument: Let $H$ be a proper subgroup of $G$ then we may always find two distinct (linear) characters of $G$ that have the same restriction on $H$. Indeed, by the previous argument there exists a conjugacy class $C$ of $G$ that does not intersect $H$. Let $D=GC$ and define $f$ to be the class function which is equal to $0$ on $D$ and $1$ on $C$ and let $g$ be the class function which is equal to $1$ everywhere.
Since $f$ and $g$ are (in a unique way) linear combinations of irreducible characters of $G$ and $fH=gH$ there must exist distinct 


A superficially different counting argument, which boils down to the same proof as before: If $H$ is a proper subgroup whose conjugates completely cover $G$, then let $G$ act on the right cosets of $H$ by right multiplication. This action is transitive. Since $H$ is a point stabilizer, the conjugates of $H$ are just all the point stabilizers. Then saying that the conjugates of $H$ cover $G$ is saying that every element of this permutation group has a fixed point. In a transitive permutation group, the average number of fixed points is $1$. The number of fixed points of the identity is the number of points, $[G:H]$. The only way every permutation can have at least the average number of fixed points is for every permutation to have exactly the average number of fixed points, so $[G:H]=1$ contradicting the assumption that $H$ is proper. 

