If you regard each row of your desired matrix as a sequence of bipolar signals and each column as a time frame, then, with one additional condition that the matrix is circulant, what you're asking becomes a set of bipolar binary sequences whose periodic out-of-phase autocorrelations are either $+1$ or $-1$. Because such sequences are close to optimal for various purposes, most likely there are a bunch of known results in the intersection of information theory, signal processing, and design theory. (The caveat is that when $n \equiv 3 \pmod{4}$, you might be able to have one more row.)

For instance, if you take the binary $m$-sequence of period $2^m-1$, its out-of-phase atuocorrelations are always $-1$. Hence, by stacking all the $2^m-1$ cyclic shifts of the sequence, you have a desired square matrix, where the inner product between a pair of rows is always $-1$.

You can exploit this idea with other well-known sequences. For example, the Legendre sequence of period $n$ has optimal autocorrelations if and only if $n \equiv 3 \pmod{4}$, i.e., if $n$ is of the form $4k-1$, its out-of-phase autocorrelations are $-1$ just like $m$-sequences. Hence, all Lengendre sequences of period $n = 4k-1$ can be turned into square matrices with the desired property.

You can use design theory with this sequence approach as well. Take a cyclic difference set $D$ of order $n$, block size $k$, and index $\lambda$. Construct the $n$-dimensional vector $\boldsymbol{a} = (a_0,\dots,a_{n-1})$, where $a_i = -1$ if $i \in D$ and $a_i = 1$ otherwise. Then, it is straightforward to check that the inner product between $\boldsymbol{a}$ and any of its cyclic shift is exactly $n-4(k-\lambda)$ (except when you take the product of exactly the same vectors). Hence, by taking cyclic difference sets satisfying $n-4(k-\lambda) = -1$ or $1$ such as the cyclic $(19,10,5)$ difference set $D = \{0,1,4,5,6,7,9,11,16,17\}$, you obtain a desired square matrix by stacking the cyclic shifts of the corresponding vector.

A good reference book for such sequences that is mathematician-friendly is Sequence Design for Communications Applications by P. Fan and M Darnell.

The second edition of Handbook of Combinatorial Designs edited by C. J. Colbourn and J. H. Dinitz has a section for "Sequence Correlation" within Chapter "Hadamard Matrices and Related Designs."

I'm not sure if you can construct an $n \times n$ or $(n+1)\times n$ almost Hadamard matrix this way when there is no known Hadamard matrix of size $n$ if you stick with sequences whose out-of-phase autocorrelations are always $-1$. But if you allow them to be either $1$ or $-1$, maybe you can for many values of $n$.

Also, this is somewhat trivial, but your method of adding/deleting columns works for partial Hadamard matrices. Fortunately, Seraj made a reference request on partial Hadamard matrix and got a nice answer by Carlo Beenakker here:

Reference for partial Hadamard matrices

This should give not necessarily circulant examples with many rows.

I know this "answer" doesn't give a satisfactory answer to your question. But searching the literature (and the internet) with some of the keywords in this post may be a good starting point.

If you regard each row of your desired matrix as a sequence of bipolar signals and each column as a time frame, then, with one additional condition that the matrix is circulant, what you're asking becomes a set of bipolar binary sequences whose periodic out-of-phase autocorrelations are either $+1$ or $-1$. Because such sequences are close to optimal for various purposes, most likely there are a bunch of known results in the intersection of information theory, signal processing, and design theory. (The caveat is that when $n \equiv 3 \pmod{4}$, you might be able to have one more row.)

For instance, if you take the binary $m$-sequence of period $2^m-1$, its out-of-phase atuocorrelations are always $-1$. Hence, by stacking all the $2^m-1$ cyclic shifts of the sequence, you have a desired square matrix, where the inner product between a pair of rows is always $-1$.

You can exploit this idea with other well-known sequences. For example, the Legendre sequence of period $n$ has optimal autocorrelations if and only if $n \equiv 3 \pmod{4}$, i.e., if $n$ is of the form $4k-1$, its out-of-phase autocorrelations are $-1$ just like $m$-sequences. Hence, all Lengendre sequences of period $n = 4k-1$ can be turned into square matrices with the desired property.

You can use design theory with this sequence approach as well. Take a cyclic difference set $D$ of order $n$, block size $k$, and index $\lambda$. Construct the $n$-dimensional vector $\boldsymbol{a} = (a_0,\dots,a_{n-1})$, where $a_i = -1$ if $i \in D$ and $a_i = 1$ otherwise. Then, it is straightforward to check that the inner product between $\boldsymbol{a}$ and any of its cyclic shift is exactly $n-4(k-\lambda)$ (except when you take the product of exactly the same vectors). Hence, by taking cyclic difference sets satisfying $n-4(k-\lambda) = -1$ or $1$ such as the cyclic $(19,10,5)$ difference set $D = \{0,1,4,5,6,7,9,11,16,17\}$, you obtain a desired square matrix by stacking the cyclic shifts of the corresponding vector.

A good reference book for such sequences that is mathematician-friendly is Sequence Design for Communications Applications by P. Fan and M Darnell.

The second edition of Handbook of Combinatorial Designs edited by C. J. Colbourn and J. H. Dinitz has a section for "Sequence Correlation" within Chapter "Hadamard Matrices and Related Designs."

I'm not sure if you can construct an $n \times n$ or $(n+1)\times n$ almost Hadamard matrix this way when there is no known Hadamard matrix of size $n$ if you stick with sequences whose out-of-phase autocorrelations are always $-1$. But if you allow them to be either $1$ or $-1$, maybe you can for many values of $n$.

Also, this is somewhat trivial, but your method of adding/deleting columns works for partial Hadamard matrices. Fortunately, Seraj made a reference request on partial Hadamard matrix and got a nice answer by Carlo Beenakker here:

Reference for partial Hadamard matrices

This should give not necessarily circulant examples with many rows.

I know this "answer" doesn't give a satisfactory answer to your question. But searching the literature (and the internet) with some of the keywords in this post may be a good starting point.

If you regard each row of your desired matrix as a sequence of bipolar signals and each column as a time frame, then, with one additional condition that the matrix is circulant, what you're asking becomes a set of bipolar binary sequences whose periodic out-of-phase autocorrelations are either $+1$ or $-1$. Because such sequences are close to optimal for various purposes, most likely there are a bunch of known results in the intersection of information theory, signal processing, and design theory.

For instance, if you take the binary $m$-sequence of period $2^m-1$, its out-of-phase atuocorrelations are always $-1$. Hence, by stacking all the $2^m-1$ cyclic shifts of the sequence, you have a desired square matrix, where the inner product between a pair of rows is always $-1$.

You can exploit this idea with other well-known sequences. For example, the Legendre sequence of period $n$ has optimal autocorrelations if and only if $n \equiv 3 \pmod{4}$, i.e., if $n$ is of the form $4k-1$, its out-of-phase autocorrelations are $-1$ just like $m$-sequences. Hence, all Lengendre sequences of period $n = 4k-1$ can be turned into square matrices with the desired property.

You can use design theory with this sequence approach as well. Take a cyclic difference set $D$ of order $n$, block size $k$, and index $\lambda$. Construct the $n$-dimensional vector $\boldsymbol{a} = (a_0,\dots,a_{n-1})$, where $a_i = -1$ if $i \in D$ and $a_i = 1$ otherwise. Then, it is straightforward to check that the inner product between $\boldsymbol{a}$ and any of its cyclic shift is exactly $n-4(k-\lambda)$ (except when you take the product of exactly the same vectors). Hence, by taking cyclic difference sets satisfying $n-4(k-\lambda) = -1$ or $1$ such as the cyclic $(19,10,5)$ difference set $D = \{0,1,4,5,6,7,9,11,16,17\}$, you obtain a desired square matrix by stacking the cyclic shifts of the corresponding vector.

A good reference book for such sequences that is mathematician-friendly is Sequence Design for Communications Applications by P. Fan and M Darnell.

The second edition of Handbook of Combinatorial Designs edited by C. J. Colbourn and J. H. Dinitz has a section for "Sequence Correlation" within Chapter "Hadamard Matrices and Related Designs."

I'm not sure if you can construct an $n \times n$ or $(n+1)\times n$ almost Hadamard matrix this way when there is no known Hadamard matrix of size $n$ if you stick with sequences whose out-of-phase autocorrelations are always $-1$. But if you allow them to be either $1$ or $-1$, maybe you can for many values of $n$.

Also, this is somewhat trivial, but your method of adding/deleting columns works for partial Hadamard matrices. Fortunately, Seraj made a reference request on partial Hadamard matrix and got a nice answer by Carlo Beenakker here:

Reference for partial Hadamard matrices

This should give not necessarily circulant examples with many rows.

I know this "answer" doesn't give a satisfactory answer to your question. But searching the literature (and the internet) with some of the keywords in this post may be a good starting point.

If you regard each row of your desired matrix as a sequence of bipolar signals and each column as a time frame, then, with one additional condition that the matrix is circulant, what you're asking becomes a set of bipolar binary sequences whose periodic out-of-phase autocorrelations are either $+1$ or $-1$. Because such sequences are close to optimal for various purposes, most likely there are a bunch of known results in the intersection of information theory, signal processing, and design theory. (The caveat is that when $n \equiv 3 \pmod{4}$, you might be able to have one more row.)

For instance, if you take the binary $m$-sequence of period $2^m-1$, its out-of-phase atuocorrelations are always $-1$. Hence, by stacking all the $2^m-1$ cyclic shifts of the sequence, you have a desired square matrix, where the inner product between a pair of rows is always $-1$.

You can exploit this idea with other well-known sequences. For example, the Legendre sequence of period $n$ has optimal autocorrelations if and only if $n \equiv 3 \pmod{4}$, i.e., if $n$ is of the form $4k-1$, its out-of-phase autocorrelations are $-1$ just like $m$-sequences. Hence, all Lengendre sequences of period $n = 4k-1$ can be turned into square matrices with the desired property.

You can use design theory with this sequence approach as well. Take a cyclic difference set $D$ of order $n$, block size $k$, and index $\lambda$. Construct the $n$-dimensional vector $\boldsymbol{a} = (a_0,\dots,a_{n-1})$, where $a_i = -1$ if $i \in D$ and $a_i = 1$ otherwise. Then, it is straightforward to check that the inner product between $\boldsymbol{a}$ and any of its cyclic shift is exactly $n-4(k-\lambda)$ (except when you take the product of exactly the same vectors). Hence, by taking cyclic difference sets satisfying $n-4(k-\lambda) = -1$ or $1$ such as the cyclic $(19,10,5)$ difference set $D = \{0,1,4,5,6,7,9,11,16,17\}$, you obtain a desired square matrix by stacking the cyclic shifts of the corresponding vector.

A good reference book for such sequences that is mathematician-friendly is Sequence Design for Communications Applications by P. Fan and M Darnell.

The second edition of Handbook of Combinatorial Designs edited by C. J. Colbourn and J. H. Dinitz has a section for "Sequence Correlation" within Chapter "Hadamard Matrices and Related Designs."

I'm not sure if you can construct an $n \times n$ or $(n+1)\times n$ almost Hadamard matrix this way when there is no known Hadamard matrix of size $n$ if you stick with sequences whose out-of-phase autocorrelations are always $-1$. But if you allow them to be either $1$ or $-1$, maybe you can for many values of $n$.

Also, this is somewhat trivial, but your method of adding/deleting columns works for partial Hadamard matrices. Fortunately, Seraj made a reference request on partial Hadamard matrix and got a nice answer by Carlo Beenakker here:

Reference for partial Hadamard matrices

This should give not necessarily circulant examples with many rows.

I know this "answer" doesn't give a satisfactory answer to your question. But searching the literature (and the internet) with some of the keywords in this post may be a good starting point.