Let $S$ denote the set of all complex nonnegative definite matrices with all diagonal elements being less that or equal to one. Can we show that any matrix which belongs to the set of all nonzero extreme points of $S$ has all its diagonal elements equal to zero or one?

2$\begingroup$ Nonnegative definite = hermitian with some eigenvalues positive? Or do you mean something else? $\endgroup$– Igor RivinApr 13, 2018 at 0:02

$\begingroup$ @Igor. Hermitian + all eigenvalues nonnegative. $\endgroup$– MathbuffApr 13, 2018 at 0:06

$\begingroup$ This cannot be true because the set is the closed convex hull of its extreme points. Or in more elementary style, $0$ is certainly an extreme point. $\endgroup$– Christian RemlingApr 13, 2018 at 0:25

$\begingroup$ What could perhaps be true is that the extreme points have diag entries zero or one. $\endgroup$– Christian RemlingApr 13, 2018 at 0:26

2$\begingroup$ Standard terminology is "positive semidefinite" (PSD). $\endgroup$– Igor RivinApr 13, 2018 at 18:13
2 Answers
Rank one matrices $xx^\top$ are extreme, now take $x=(1,1/2)$. This gives $\begin{pmatrix} 1&1/2\\1/2&1/4\end{pmatrix}$, a counterexample to your conjecture.

$\begingroup$ @ Dima and Igor. I am talking about extreme points of the set of all nonnegative definite matrices. I am considering a special convex set of nonnegative matrices with diagonal entries all less than or equal to one. So your argument is not correct. $\endgroup$– MathbuffApr 14, 2018 at 12:26

2$\begingroup$ the matrix I give is in your set, and we claim it is extreme $\endgroup$ Apr 14, 2018 at 12:36

$\begingroup$ @ Dima and Igor. I am really sorry. you are absolutely right. $\endgroup$– MathbuffApr 14, 2018 at 14:03
The extremal rays of the PSD cone are the rank one matrices (also known as projections),so those of the form $x x^t.$  see, for example, https://math.stackexchange.com/questions/678693/positivesemidefiniteconeisgeneratedbyallrankonematrices
The answer to your question follows immediately.

1$\begingroup$ no, the question is whether on the boundary one always get matrices with only 0 or 1 on the diagonal. E.g. for x=(1,1/2) one gets an extreme matrix which is not like this. $\endgroup$ Apr 13, 2018 at 19:33