Here is a partial positive answer, going in the opposite direction of Terry´s comment.

Let $X$ be an infinite topological space. Suppose $X$ is first countable at $p \in X$ and for every open neighborhood $U$ of $p$ there is a smaller neighborhood $V \subseteq U$ such that $|U \setminus V|=|U|$. Then there is a group structure on $X$ with identity $p$ such that the operation $(x,y) \mapsto xy^{-1}$ is continuous at $(p,p)$.

Just note that (starting with a $U_0$ of minimal cardinality) we can find $\{U_n : n \in \omega\}$ an open local base at $p$ such that for every $n \in \omega$ we have $U_{n+1} \subset U_n$ and $|U_n \setminus U_{n+1}|=|U_0|$. Now find (e.g. using compactness of first-order logic) a group $G$ containing subgroups $H_0 \supset H_1 \supset H_2 \supset \cdots$, such that $|G|=|X|$ and $|H_n \setminus H_{n+1}|=|H_0|=|U_0|$ for every $n \in \omega$, and then transfer the group structure to $X$ in the obvious way.

Note that we can also change the hypothesis to: there is an open neighborhood of $p$ with no smaller neighborhoods (e.g. $p$ is isolated) and the conclusion still holds; just call $U_0$ such neighborhood, find $G$ and $H_0$ as before and ignore the rest.

Here is a partial positive answer, going in the opposite direction of Terry´s comment.

Let $X$ be an infinite topological space. Suppose $X$ is first countable at $p \in X$ and for every open neighborhood $U$ of $p$ there is a smaller neighborhood $V \subseteq U$ such that $|U \setminus V|=|U|$. Then there is a group structure on $X$ with identity $p$ such that the operation $(x,y) \mapsto xy^{-1}$ is continuous at $(p,p)$.

Just note that (starting with a $U_0$ of minimal cardinality) we can find $\{U_n : n \in \omega\}$ an open local base at $p$ such that for every $n \in \omega$ we have $U_{n+1} \subset U_n$ and $|U_n \setminus U_{n+1}|=|U_0|$. Now find (e.g. using compactness of first-order logic) a group $G$ containing subgroups $H_0 \supset H_1 \supset H_2 \supset \cdots$, such that $|G|=|X|$ and $|H_n \setminus H_{n+1}|=|H_0|=|U_0|$ for every $n \in \omega$, and then transfer the group structure to $X$ in the obvious way.

Note that we can also change the hypothesis to: there is an open neighborhood of $p$ with no smaller neighborhoods (e.g. $p$ is isolated) and the conclusion still holds; just call $U_0$ such neighborhood, find $G$ and $H_0$ as before and ignore the rest.

Edit (Some details as to why such $G$ exists): Let $\kappa=|X|$ and $\mu=|U_0|$. Consider the language of first order logic consisting of a binary function symbol (for the group operation), countably many unary predicate symbols $\{P_n : n \in \omega\}$ and constant symbols $\{c^n_\alpha : \alpha \in \mu, n \in \omega\}$. In this language consider the first order theory that includes the group axioms, for each $n$ the axioms saying that $P_n$ is a subgroup and $P_{n+1} \subseteq P_n$, axioms saying that all the $c^n_\alpha$'s are distinct and $c^n_\alpha \in P_{n} \setminus P_{n+1}$. This theory is consistent because each finite fragment of it can be satisfied, e.g. using $\mathbb{Z}$ and finitely many of its subgroups. Since the language has size $\mu$, there is (by Lowenheim-Skolem) a model for this theory of size $\mu$. Inside this model we can find our $H_n$´s as the interpretations of the $P_n$'s. Finally we let $G=H_0 \times K$ where $K$ is any group of size $\kappa$.