Finite non-commutative ring with few invertible (unit) elements for a ring $R$ with unity , let $U(R)$ denote the group of units of $R$ . Now there are lots of finite commutative rings, of arbitrarily high order,  with exactly one unit ; indeed $U(R)=1$ for a finite commutative ring $R$ iff $a^2=a , \forall a \in R$ . Incidentally , I couldn't find any finite non-commutative ring with exactly one unit; matrix rings don't seem to work.
So my question is : Does there exist a finite non-commutative ring with unity having exactly one invertible (unit)  element ?
Small remark : Note that such a ring must have characteristic $2$
 A: I think I have it ; there can be no such non-commutative ring. 
Let $x \in J(R)$ , then $1-x$ is a unit of $R$ , so $x=0$ i.e. $J(R)=0$ . Thus $R$ is an artinian semisimple ring , so by Artin-Wedderburn , $R \cong \prod_{i=1}^m M(n_i , D_i) $ , where $D_i$'s are division rings . But $R$ is finite , hence so are  $D_i$'s , hence by Wedderburn little theorem, $D_i$'s are fields , so $R \cong \prod_{i=1}^m M(n_i , k_i) $ , where $k_i$'s are fields . Now since $R$ is not-commutative , at least one $n_i$ is more than $1$ , say w.l.o.g. $n_1 \ge 2$ , but then $M(n_1 , k_1)$ has at least $q^{n_1}-1 \ge q^2-1 >1$ many units (where $q=|k_1|$) , so $R$ has more than one unit . 
In fact , since $M(n_1,k_1)$ has $\prod_{j=0}^{n_1-1}(q^{n_1} - q^j)$ many units , where $q=|k_1|$ and for $n_1 \ge 2$ , $\prod_{j=0}^{n_1-1}(q^{n_1} - q^j)\ge (2^2-1)(2^2-2)=6$ ; so we get that : 
Any finite non-commutative ring with unity and with zero Jacobson radical has at least $6$ units  .
A: This answer presents an alternate proof  of users' negative answer by proving directly that a finite ring whose only unit is its identity must be a Boolean ring, hence commutative. The proof given below is based on a result by Melvin Henriksen. It doesn't rely on the Artin-Wedderburn Theorem and turns out to be fully elementary.
Following Melvin Henriksen, we call $R$ a UI-ring if $R$ has an identity element $1$ and $ab = ba = 1$ for $a,b \in R$ implies $a = b = 1$.
We have


Claim. A finite ring $R$ with identity is a UI-ring if and only if $R$ consists only of idempotent elements, i.e., $R$ is a Boolean ring. In particular, a finite UI-ring is commutative.
Proof. Assume that $R$ is a UI-ring. Then $R$ is reduced and $2x = 0$ for every $x \in R$. As $R$ is a finite dimensional vector space over $\mathbb{Z}/2\mathbb{Z}$, every element of $R$ is algebraic over $\mathbb{Z}/2\mathbb{Z}$ by the Cayley-Hamilton theorem. Thus 
    $R$ is a Boolean ring by [2, Corollary 2.10], which shows that $R$ is commutative. Assume now that $R$ is a Boolean ring. As any element $x \neq 1$ satisfies $x(1 - x) = 0$, the identity $1$ is the only unit of $R$.


The commutative case mentioned in OP's question was solved by P. M. Cohn [2, Theorem 3], should $R$ be finite or infinite:


Cohn's Theorem 
    Let $R$ be an algebra over a field $F$ without nontrivial units, i.e., the units of $R$ are those of $F$. Then
    $R$ is a subdirect product of extension fields of $F$, and every element $x$ of $R$ which is not in $F$ is transcendental over $F$, unless $F = GF(2)$ and $x$ is idempotent. If, moreover, $R$ has finite dimension over $F$, then either $R = F$ or $R$ is a Boolean algebra.


Addendum. I discovered this preprint of Rodney Coleman (2013) in which OP's question was both asked and answered.

[1] P. M. Cohn, "Rings of zero-divisors", 1984.
[2] M. Henriksen, "Rings with a unique regular element", 1989.   
A: As mentioned by Luc Guyot in his answer (and found independently in another thread), unital rings with the property that every element other than the identity is a zero divisor, were first studied by P.M. Cohn (though only in the commutative case) in

*

*Rings of zero divisors, Proc. Amer. Math. Soc. 9 (1958), 914-919.

Rings from this class are commonly referred to as "$0$-rings" or "$\mathcal O$-rings", and it is proved by H.G. Moore, S.J. Pierce, and A. Yaqub in

*

*Commutativity in rings of zero divisors, Amer. Math. Monthly 75 (1968), 392

that every right (or left) artinian $\mathcal O$-ring is a Boolean ring (and hence commutative). This answers a stronger version of the question asked in the OP.
In fact, every finite ring $R$ is, of course, (left and right) artinian; and if the unique unit of $R$ is the identity $1_R$, then $R$ is necessarily an $\mathcal O$-ring (for every $x \in R$, there exist $m, n \in \mathbf N$ with $m < n$ such that $x^m = x^n$, implying that, if $x$ is left- or right-cancellative, then $x^{n-m} = 1_R$ and hence $x \in R^\times$).
