$\DeclareMathOperator\Pic{Pic}\DeclareMathOperator\GL{GL}\DeclareMathOperator\GO{GO}$
Let $G$ be a connected linear algebraic group over an algebraically closed field $K$ of characteristic 0.
Let $F\subseteq G$ be an algebraic $K$-subgroup,
not necessarily connected, and set $Y=G/F$.
Then there is a canonical isomorphism
$$
\operatorname{coker}\left[{\sf X}^*(G)\to {\sf X}^*(F)\right]\overset{\lambda}{\longrightarrow}\Pic Y,
$$
where ${\sf X}^*(G)$ denotes the character group of $G$,
and the map ${\sf X}^*(G)\to {\sf X}^*(F)$ is the restriction homomorphism.
This follows, for instance, from Theorem 2 in the introduction of
M. Borovoi and J. van Hamel, Extended equivariant Picard complexes and homogeneous spaces, Transform. Groups 17 (2012), 51-86, arXiv:1010.3414.
The map $\lambda$ sends a character $\chi\in{\sf X}^*(F)$ to the class of the ${\Bbb G}_m$-torsor $Y_\chi\to Y$,
where $Y_\chi=(G\times {\Bbb G}_m)/\chi_*(F)$ and
$$\chi_*\colon F\to G\times {\Bbb G}_m,\quad f\mapsto(f,\chi(f))\ \text{ for }f\in F.$$
In our case
$$ F=\left\{
\begin{pmatrix}
A&B\\0&D
\end{pmatrix}
\ \ \Big |\ \ A\in\GO_k,\ B\in{\rm Mat}_{k,\,n-k}, D\in \GL_{n-k}
\right\}.
$$
Clearly,
$${\sf X}^*(F)={\sf X}^*(\GO_k)\oplus {\sf X}^*(\GL_{n-k}).$$
If $k<n$, we have ${\sf X}^*(\GL_{n-k})\cong {\Bbb Z}$ with generator $\det_{n-k}$.
(If $k=n$, then of course ${\sf X}^*(\GL_{n-k})=0$.)
We write $X:={\sf X}^*(\GO_k)$.
For $k=1$ we have $X\simeq {\Bbb Z}$.
For $2\le k\le n$, the group $X$ is generated by $d=\det_k$ and $c$ with one relation $2d-kc=0$.
In other words,
$$X\cong{\Bbb Z}^2/\langle (2,-k)\rangle.$$
If $k$ is odd, $k=2p+1$, then the element $(2, -k)\in{\Bbb Z}^2$ is primitive (indivisible),
and hence the group $X$ is cyclic.
Namely, we consider the following basis of ${\Bbb Z}^2$:
$$e_1=2d-(2p+1)c,\quad e_2=d-pc; $$
then $X\cong {\Bbb Z}^2/\langle e_1\rangle \simeq {\Bbb Z}$ with a generator of infinite order $[e_2]$.
If $k$ is even, $k=2p$, then the element $(2, -k)=2(1, -p)\in{\Bbb Z}^2$ is divisible by 2,
and hence the group $X$ is isomorphic to ${\Bbb Z}/2{\Bbb Z}\oplus {\Bbb Z}$.
Namely, we consider the following basis of ${\Bbb Z}^2$:
$$e_1=d-pc,\quad e_2=c;$$
then $X\cong{\Bbb Z}^2/\langle 2e_1\rangle\simeq {\Bbb Z}/2{\Bbb Z}\oplus{\Bbb Z}$
with a generator $[e_1]$ of order 2 and a generator $[e_2]=[c]$ of infinite order.
We assume that $n\ge 2$. If $k<n$, the map ${\sf X}^*(G)\to {\sf X}^*(\GL_{n-k})$ is bijective, and hence
$$\operatorname{coker}\left[{\sf X}^*(G)\to {\sf X}^*(F)\right]\simeq {\sf X}^*(\GO_k)=X.$$
Thus for $1\le k<n$
$$
\Pic Y\simeq
\begin{cases}
{\Bbb Z} &\text{if $k$ is odd;}\\
{\Bbb Z}/2{\Bbb Z}\oplus {\Bbb Z} &\text{if $k$ is even.}
\end{cases}
$$
For $k=n$ we have ${\sf X}^*(F)=X$ and
$$\Pic Y\cong \operatorname{coker}\left[{\sf X}^*(G)\to X\right].$$
Now
$$
\operatorname{coker}\left[{\sf X}^*(G)\to X\right]={\Bbb Z}^2/\langle (1,0),(2,-n)\rangle\simeq {\Bbb Z}/n{\Bbb Z}
$$
with the generator $[c]$ of order $n$.
EDIT:
Our answers for $k=1$ and $k=n$ coincide with those of OP, but not for $k=2$, $n=3$.
Below we compute ${\sf X}^*(F)$ and $\operatorname{coker}[{\sf X}^*(G)\to{\sf X}^*(F)]$ is the case $k=2$, $n>2$.
Recall that
$$\GO_2=\{A\in\GL_2\mid AA^T=c(A) I_2\}.$$
Elementary calculations show that
$$ \GO_2=\{U(a,b), V(a,b)\mid a^2+b^2\neq 0\},$$
where
$$
U(a,b)=\begin{pmatrix}a&b\\-b&a\end{pmatrix},\quad V(a,b)=\begin{pmatrix}a&b\\b&-a\end{pmatrix}.
$$
We have
$$U(a,b)\cdot U(a,-b)=(a^2+b^2)I_2$$
whence
$$ U(a,b)^{-1}=U(a,-b)/(a^2+b^2).$$
Set
$$ U=\{U(a,b)\},\quad V=\{V(a,b)\},\quad v=V(1,0)={\rm diag}(1,-1).$$
Then $U$ is a subgroup of $\GO_2$, and $\GO_2=U\cup vU$.
The group $\GO_2$ is not abelian. Indeed,
$$ vU(a,b)v^{-1}U(a,b)^{-1}=U(a,-b)\cdot U(a,b)^{-1}=U(a,-b)^2/(a^2+b^2).$$
We obtain that the commutator subgroup
$$(\GO_2,\GO_2)=U_1:=\{U(a,b)\mid a^2+b^2=1\}.$$
It follows that ${\sf X}^*(U)\cong {\Bbb Z}$ with generator $\omega$ given by $\omega(U(a,b))=a^2+b^2$.
Then
$\omega=d|_U=c|U$ and $\ker\omega=U_1$.
We have an isomorphism
$$\omega_*\colon U/U_1\to {\Bbb G}_m,\quad U(a,b)\cdot U_1\mapsto a^2+b^2.$$
The map
$$ U\times\{1,v\}\to \GO_2,\quad (U(a,b), 1)\mapsto U(a,b), \ \, (U(a,b),v)\mapsto vU(a,b)=V(a,b)$$
induces an isomorphism
$$U/U_1\times \{1,v\}\overset\sim\longrightarrow \GO_2/U_1.$$
Thus $\GO_2/(\GO_2,\GO_2)\cong {\Bbb G}_m\times {\Bbb Z}/2{\Bbb Z}$,
and ${\sf X}^*(\GO_2)\cong {\Bbb Z}\oplus{\Bbb Z}/2{\Bbb Z}$.
We have
\begin{align*}&d(U(a,b))=c(U(a,b))=a^2+b^2,\\
&c(V(a,b))=a^2+b^2,\ \text{ but }\ d(V(a,b))=-(a^2+b^2).
\end{align*}
Thus the character $\zeta:=d/c$ of $\GO_2$ takes the value 1 on $U$ and the value $-1$ on $V$.
Clearly, $\zeta\neq 1$, but $\zeta^2=1$. Thus the character $\zeta$ is of order 2.
Clearly, ${\sf X}^*(\GO_2)\cong {\Bbb Z}\oplus {\Bbb Z}/2{\Bbb Z}$ with generator $d$ of infinite order
and generator $\zeta$ of order 2.
Now we assume that $n>2$. We compute $\operatorname{coker}[{\sf X}^*(G)\to{\sf X}^*(F)]$.
We have ${\sf X}^*(G)\cong {\Bbb Z}$ with generator $d_n$, and
$${\sf X}^*(F)={\sf X}^*(\GL_{n-k})\oplus{\sf X}^*(\GO_2)\cong {\Bbb Z}\oplus{\Bbb Z}\oplus{\Bbb Z}/2{\Bbb Z}$$
with generators $d_{n-k}$, $d=d_2$, and $\zeta$.
The restriction map ${\sf X}^*(G)\to{\sf X}^*(F)$ sends $d_n$ to $(d_{n-k},d_2,0)$.
It follows that $\operatorname{coker}[{\sf X}^*(G)\to{\sf X}^*(F)]\simeq {\Bbb Z}\oplus{\Bbb Z}/2{\Bbb Z}$
with generator $[d_2]$ of infinite order and generator $[\zeta]$ of order 2.
Thus $\Pic Y\simeq {\Bbb Z}\oplus{\Bbb Z}/2{\Bbb Z}$.
I construct explicitly a ${\Bbb G}_m$-torsor over $Y$ of order 2 in the Picard group.
We consider the homomorphism
$$\zeta_*\colon F\to G\times{\Bbb G}_m,\quad f\mapsto (f,\zeta(f))\,\text{ for }f\in F,$$
where we extend the character $\zeta$ of $\GO_2$ to $F$ by
$$
\zeta\begin{pmatrix}
A&B\\0&D
\end{pmatrix}
:=\zeta(A).
$$
We consider the quotient
$$ Y_\zeta=(G\times{\Bbb G}_m)/\zeta_*(F)$$
and the projection map
$$\pi\colon Y_\zeta\to Y,\quad (g,z)\cdot \zeta_*(F)\mapsto g\cdot F.$$
Then the class in $\Pic Y$ of the ${\Bbb G}_m$-torsor $(Y_\zeta,\pi)$ is of order 2.
\operatorname{GL}(n)in the title, or even$\DeclareMathOperator\GL{GL}\GL(n)$. $\endgroup$