See Frege's Theorem: First Derivation of the Contradiction for reference.
Let "$P$" abbreviate the name for the concept being the extension of a concept which you don't fall under:
$[λx ∃F(x = εF \land ¬Fx)]$.
In Grundgesetze's logic it is proved that $εP$ exists, where for each value of the variable $F$, $εF$ denotes the extension of $F$.
Now supposing $P(εP)$, by Basic Law V, we have $¬P(εP)$, contrary to hypothesis.
Thus, from $(\varphi \to \lnot \varphi) \to \lnot \varphi$, we have: $\lnot P(εP)$. But then we have that $P(εP)$, again contrary to hypothesis.
Having derived a contradiction (which is a purely syntactical fact), we have to reject (as done by Frege) Basic Law V, i.e. to conclude with $\vdash \lnot \text { Basic Law V}$.
This means that, using Frege's "horizontal" fucntion $-$:
$- \Delta =$ the True if $\Delta$ is the True; the False otherwise,
we have that:
$- \text { Basic Law V}$ denotes the False.
See Frege's letter to Russell (1902) on the contradiction:
Your discovery of the contradiction caused me the greatest surprise and, I would almost say, consternation, since it has shaken the basis on which I intended to build arithmetic. It seems, then, that transforming the generalization of an equality into an equality of courses-of-values is not always permitted, that my Rule V is false, and that my explanations are not sufficient to ensure that my combinations of signs have a meaning in all cases [emphasis added].
This can be interpreted as meaning that it is not true in general that:
for each value of the variable $F$, the extension of $F$: $εF$, exists.
And this in turn implies that the "combinations of signs" $εF$ is not always meaningful.
But note also Frege's clarification of the "form" of the contradiction in Grundgesetze's system:
Incidentally, it seems to me that the expression "a predicate is predicated of itself" is not exact. A predicate is as a rule a first-level function, and this function requires an object as argument and cannot have itself as argument (subject). Therefore I would prefer to say "a notion is predicated of its own extension". If the function $\Phi(\xi)$ is a concept, I denote its extension (or the corresponding class) by "$ἐ \Phi(ε)$". In "$\Phi(ἐ \Phi(ε))$" we then have a case in which the concept $\Phi(\xi)$ is predicated of its own extension.
See On Sense and Reference (1892) :
[page 32] Is it possible that a sentence as a whole has only a sense, but no meaning? At any rate, one might expect that such sentences occur, just as there are parts of sentences having sense but no meaning. And sentences which contain proper names without meaning will be of this kind.
[page 41] This arises from an imperfection of language, from which even the symbolic language of mathematical analysis is not altogether free; even there combinations of symbols can occur that seem to mean something but (at least so far) do not mean anything, e.g. divergent infinite series.
[...] A logically perfect language (Begriffsschrift) should satisfy the conditions, that every expression grammatically well constructed as a proper name out of signs already introduced shall in fact designate an object, and that no new sign shall be introduced as a proper name without being secured a
meaning. The logic books contain warnings against logical mistakes
arising from the ambiguity of expressions. I regard as no less pertinent
a warning against apparent proper names without any meaning.
So, if the set $R$ does not exist, the name for it lacks reference, and thus any sentence containing it lacks truth-value.
But in a logically perfect language every expression grammatically well constructed must have reference: an object for a name and a truth-value for a sentence.
Conclusion: R \notin R$ is not "an expression grammatically well constructed" in Frege's system.
Frege's first (and unsuccessful) attempt to salvage his system was precisely to restric the application of Basic Law V (which does not apply to propositions, but to functions: $ἐf(ε) = ἀg(α) ≡ ∀x[f(x) = g(x)]$ ); see:
It may be interesting to consider that, in his "modified" system sketched into the Appendix to GG II, Frege claims to have proved, with its modification to BL V, that:
$\vdash \lnot x \text ^ x$
which amounts (approximately) to: $\vdash R \notin R$.
Regarding Klement's paper, it discuss a very subtle point, which supports your previous concern that Basic Law V is (presumably) not the only source of trouble in Frege's system.
As Klement notes:
A few months after reporting Russell’s paradox, Russell despaired in a letter to Frege that "from Cantor’s proposition that any class contains more subclasses than objects we can elicit constantly new contradictions", and reports on his discovery of a Cantorian paradox regarding propositions, suggesting that, mutatis mutandis, this should be a worry for Frege’s theory of thoughts. In the
ensuing correspondence, Russell formulates the paradox in terms of both
generating a distinct proposition for every class, as well as generating a
distinct proposition for every propositional function. Frege never quite
fully appreciated Russell’s point, but nevertheless Russell had put his
finger on a genuine issue Frege should have been concerned with, as I have
argued elsewhere.
If we apply modern semantics to Frege's system, we have that every concept "identify" a subset of the universe. Thus, by Cantor's theorem, there are more concepts than objects in the universe.
But in Frege's system - and this assumption is prior to Basic Law V - for every concept must exists its extension, and extensions are objects; thus, we have already an inconsistency, as suggested by Russell.
