$\DeclareMathOperator\Pic{Pic}\DeclareMathOperator\GL{GL}\DeclareMathOperator\GO{GO}$
Let $G$  be a connected reductive group over a field $K$ of characteristic 0. Let $F\subseteq G$ be a *connected* algebraic subgroup, and set $Y=G/F$.
By Proposition 6.10 in Sansuc's paper [Groupe de Brauer et arithmétique des groupes algébriques linéaires sur un corps de nombres](http://www.digizeitschriften.de/dms/img/?PID=GDZPPN002198746&physid=phys48#navi)
there is a natural exact sequence of abelian groups
$$
{\sf X}^*(G)\to {\sf X}^*(F)\overset{\lambda}{\longrightarrow} \Pic Y\to \Pic G,
$$
where ${\sf X}^*(G)$ denotes the character group of $G$,
and the map ${\sf X}^*(G)\to {\sf X}^*(F)$ is the restriction homomorphism.
In our case $G=\operatorname{GL}_n$, and hence $\Pic G=0$.
We obtain a canonical isomorphism
$$
\Pic Y\cong \operatorname{coker}\left[{\sf X}^*(G)\to {\sf X}^*(F)\right].
$$
We have a similar isomorphism in the case when $F$ is not necessarily connected.
The map $\lambda$ takes a character $\chi\in{\sf X}^*(F)$ to the class of the ${\Bbb G}_m$-torsor $Y_\chi\to Y$,
where $Y_\chi=(G\times {\Bbb G}_m)/\chi_*(F)$ and 
$$\chi_*\colon F\to G\times {\Bbb G}_m,\quad f\mapsto(f,\chi(f))\ \text{ for }f\in F.$$

In our case
$$ F=\left\{
\begin{pmatrix}
A&B\\0&D
\end{pmatrix}
\ \ \Big |\ \ A\in\GO_k,\ B\in{\rm Mat}_{k,\,n-k}, D\in \GL_{n-k}
\right\}.
$$
Clearly,
$${\sf X}^*(F)={\sf X}^*(\GO_k)\oplus {\sf X}^*(\GL_{n-k}).$$
If $k<n$, we have ${\sf X}^*(\GL_{n-k})\cong {\Bbb Z}$ with generator $\det_{n-k}$.
(If $k=n$, then of course ${\sf X}^*(\GL_{n-k})=0$.)

We write $X:={\sf X}^*(\GO_k)$.
For $k=1$ we have $X\simeq {\Bbb Z}$.
For $2\le k\le n$, the group $X$ is generated by  $d=\det_k$  and $c$ with one relation $2d-kc=0$.
In other words,
$$X\cong{\Bbb Z}^2/\langle (2,-k)\rangle.$$

If $k$ is odd, $k=2p+1$, then the element $(2, -k)\in{\Bbb Z}^2$ is primitive (indivisible),
and hence the group $X$ is cyclic. 
Namely, we consider the following basis of ${\Bbb Z}^2$:
$$e_1=2d-(2p+1)c,\quad e_2=d-pc; $$
then $X\cong {\Bbb Z}^2/\langle e_1\rangle \simeq {\Bbb Z}$ with a generator of infinite order $[e_2]$.

If $k$ is even, $k=2p$, then the element  $(2, -k)=2(1, -p)\in{\Bbb Z}^2$ is divisible by 2,
and hence the group $X$ is isomorphic to ${\Bbb Z}/2{\Bbb Z}\oplus {\Bbb Z}$.
Namely, we consider the following basis of ${\Bbb Z}^2$:
$$e_1=d-pc,\quad e_2=c;$$
then $X\cong{\Bbb Z}^2/\langle 2e_1\rangle\simeq {\Bbb Z}/2{\Bbb Z}\oplus{\Bbb Z}$
with a generator $[e_1]$ of order 2 and a generator $[e_2]=[c]$ of infinite order.

We assume that $n\ge 2$. If $k<n$, the map ${\sf X}^*(G)\to {\sf X}^*(\GL_{n-k})$ is bijective, and hence
$$\operatorname{coker}\left[{\sf X}^*(G)\to {\sf X}^*(F)\right]\simeq {\sf X}^*(\GO_k)=X.$$
Thus for $1\le k<n$
$$
\Pic Y\simeq 
\begin{cases}
{\Bbb Z}               &\text{if $k$ is odd;}\\
{\Bbb Z}/2{\Bbb Z}\oplus {\Bbb Z}  &\text{if $k$ is even.}
\end{cases}
$$

For $k=n$ we have ${\sf X}^*(F)=X$ and 
$$\Pic Y\cong \operatorname{coker}\left[{\sf X}^*(G)\to X\right].$$
Now 
$$
\operatorname{coker}\left[{\sf X}^*(G)\to X\right]={\Bbb Z}^2/\langle (1,0),(2,-n)\rangle\simeq {\Bbb Z}/n{\Bbb Z}
$$
with the generator $[c]$ of order $n$.

Our answers for $k=1$ and $k=n$ coincide with those of OP.