The notion exists both for (ordinary) groups and for algebraic groups, which are groups that are simultaneously algebraic varieties in a compatible way. Over fields that are not algebraically closed, the modern way to deal with these uses scheme-theory, which is probably beyond undergraduate mathematics. So let me focus on subgroups of the abstract group $\operatorname{GL}_n(k)$ instead of the algebraic group $\operatorname{GL}_{n,k}$.
Definition. Let $k$ be a field, and $G \subseteq \operatorname{GL}_n(k)$ a subgroup. Then $G$ is irreducible if the representation $G \to \operatorname{GL}_n(k)$ is irreducible. In other words, there is no subspace $V \subseteq k^n$ such that $g V \subseteq V$ for all $g \in G$, except $V = 0$ or $V = k^n$.
The other definition you often find is in terms of parabolic subgroups:
Definition. A subgroup $P \subseteq \operatorname{GL}_n(k)$ is parabolic if there is a flag $0 = V_0 \subsetneq \ldots \subsetneq V_r = k^n$ of subspaces for which $P$ is the stabiliser, i.e. $gV_i = V_i$ for all $i$ if and only if $g \in P$.
Then these notions are closely related:
Lemma. Let $G \subseteq \operatorname{GL}_n(k)$ be a subgroup. Then $G$ is irreducible if and only if the only parabolic subgroup $P \subseteq \operatorname{GL}_n(k)$ containing $G$ is $P = \operatorname{GL}_n(k)$.
Proof. If $G \subseteq P$ for a parabolic subgroup $P \subsetneq \operatorname{GL}_n(k)$ that is the stabiliser of some flag $0 = V_0 \subsetneq \ldots \subsetneq V_r = k^n$, then we must have $r \geq 2$ as $P \neq \operatorname{GL}_n(k)$. Then $V_1$ is a nontrivial $G$-subrepresentation of $k^n$, hence $G$ is not irreducible.
Conversely, if $G$ is not irreducible, there exists some subspace $V \subseteq k^n$ different from $0$ and $k^n$ such that $gV = V$ for all $g \in G$. Then the stabiliser of the flag $0 \subsetneq V \subsetneq k^n$ is a parabolic subgroup $P \subsetneq \operatorname{GL}_n(k)$ containing $G$. $\square$
Example. For concreteness' sake, you should probably compute some stabilisers. For example, the stabiliser in $\operatorname{GL}_2(k)$ of $V = \operatorname{span} {1 \choose 0} \subseteq k^2$ is the group of upper triangular matrices. In general, parabolic groups are conjugate to the block upper triangular form:
$$\left(\begin{array}{c|c|c} * & * & * \\\hline 0 & * & * \\\hline 0 & 0 & *\end{array}\right),$$
where the blocks on the diagonal are square (but not necessarily the others). Hint: choose a basis $v_1,\ldots,v_n$ of $k^n$ such that $V_i = \operatorname{span}(v_1,\ldots, v_{d_i})$ for $d_i = \dim V_i$.
Remark. As for deciding irreducibility, you have to do some work. Usually you show that a representation $V$ is irreducible by checking that for any nonzero vector $v \in V$, the conjugates $\{gv\ |\ g \in G\}$ span $V$. For example, the permutation matrices $S_n \subseteq \operatorname{GL}_n(k)$ are never irreducible, as $V = \operatorname{span}\Big(\begin{smallmatrix}1 \\ \vdots \\ 1\end{smallmatrix}\Big)$ is fixed by $S_n$. But $\{(x_1,\ldots,x_n)\ |\ \sum x_i = 0\} \subseteq k^n$ is irreducible as $S_n$-representation, at least when $n$ is invertible in $k$ (i.e. $\operatorname{char} k = 0$ or $\operatorname{char} k = p > 0$ and $p \nmid n$). (Exercise!)
