$\DeclareMathOperator\Pic{Pic}\DeclareMathOperator\GL{GL}\DeclareMathOperator\GO{GO}$ Let $G$ be a connected linear algebraic group over an algebraically closed field $K$ of characteristic 0. Let $F\subseteq G$ be an algebraic $K$-subgroup, not necessarily connected, and set $Y=G/F$. Then there is a canonical isomorphism $$ \operatorname{coker}\left[{\sf X}^*(G)\to {\sf X}^*(F)\right]\overset{\lambda}{\longrightarrow}\Pic Y, $$ where ${\sf X}^*(G)$ denotes the character group of $G$, and the map ${\sf X}^*(G)\to {\sf X}^*(F)$ is the restriction homomorphism. This follows, for instance, from Theorem 2 in the introduction of M. Borovoi and J. van Hamel, Extended equivariant Picard complexes and homogeneous spaces, Transform. Groups 17 (2012), 51-86, arXiv:1010.3414. The map $\lambda$ sends a character $\chi\in{\sf X}^*(F)$ to the class of the ${\Bbb G}_m$-torsor $Y_\chi\to Y$, where $Y_\chi=(G\times {\Bbb G}_m)/\chi_*(F)$ and $$\chi_*\colon F\to G\times {\Bbb G}_m,\quad f\mapsto(f,\chi(f))\ \text{ for }f\in F.$$
In our case $$ F=\left\{ \begin{pmatrix} A&B\\0&D \end{pmatrix} \ \ \Big |\ \ A\in\GO_k,\ B\in{\rm Mat}_{k,\,n-k}, D\in \GL_{n-k} \right\}. $$ Clearly, $${\sf X}^*(F)={\sf X}^*(\GO_k)\oplus {\sf X}^*(\GL_{n-k}).$$ If $k<n$, we have ${\sf X}^*(\GL_{n-k})\cong {\Bbb Z}$ with generator $\det_{n-k}$. (If $k=n$, then of course ${\sf X}^*(\GL_{n-k})=0$.)
We write $X:={\sf X}^*(\GO_k)$. For $k=1$ we have $X\simeq {\Bbb Z}$. For $2\le k\le n$, the group $X$ is generated by $d=\det_k$ and $c$ with one relation $2d-kc=0$. In other words, $$X\cong{\Bbb Z}^2/\langle (2,-k)\rangle.$$
If $k$ is odd, $k=2p+1$, then the element $(2, -k)\in{\Bbb Z}^2$ is primitive (indivisible), and hence the group $X$ is cyclic. Namely, we consider the following basis of ${\Bbb Z}^2$: $$e_1=2d-(2p+1)c,\quad e_2=d-pc; $$ then $X\cong {\Bbb Z}^2/\langle e_1\rangle \simeq {\Bbb Z}$ with a generator of infinite order $[e_2]$.
If $k$ is even, $k=2p$, then the element $(2, -k)=2(1, -p)\in{\Bbb Z}^2$ is divisible by 2, and hence the group $X$ is isomorphic to ${\Bbb Z}/2{\Bbb Z}\oplus {\Bbb Z}$. Namely, we consider the following basis of ${\Bbb Z}^2$: $$e_1=d-pc,\quad e_2=c;$$ then $X\cong{\Bbb Z}^2/\langle 2e_1\rangle\simeq {\Bbb Z}/2{\Bbb Z}\oplus{\Bbb Z}$ with a generator $[e_1]$ of order 2 and a generator $[e_2]=[c]$ of infinite order.
We assume that $n\ge 2$. If $k<n$, the map ${\sf X}^*(G)\to {\sf X}^*(\GL_{n-k})$ is bijective, and hence $$\operatorname{coker}\left[{\sf X}^*(G)\to {\sf X}^*(F)\right]\simeq {\sf X}^*(\GO_k)=X.$$ Thus for $1\le k<n$ $$ \Pic Y\simeq \begin{cases} {\Bbb Z} &\text{if $k$ is odd;}\\ {\Bbb Z}/2{\Bbb Z}\oplus {\Bbb Z} &\text{if $k$ is even.} \end{cases} $$
For $k=n$ we have ${\sf X}^*(F)=X$ and $$\Pic Y\cong \operatorname{coker}\left[{\sf X}^*(G)\to X\right].$$ Now $$ \operatorname{coker}\left[{\sf X}^*(G)\to X\right]={\Bbb Z}^2/\langle (1,0),(2,-n)\rangle\simeq {\Bbb Z}/n{\Bbb Z} $$ with the generator $[c]$ of order $n$.
EDIT: Our answers for $k=1$ and $k=n$ coincide with those of OP, but not for $k=2$, $n=3$. Below we compute ${\sf X}^*(F)$ and $\operatorname{coker}[{\sf X}^*(G)\to{\sf X}^*(F)]$ is the case $k=2$, $n>2$.
Recall that $$\GO_2=\{A\in\GL_2\mid AA^T=c(A) I_2\}.$$ Elementary calculations show that $$ \GO_2=\{U(a,b), V(a,b)\mid a^2+b^2\neq 0\},$$ where $$ U(a,b)=\begin{pmatrix}a&b\\-b&a\end{pmatrix},\quad V(a,b)=\begin{pmatrix}a&b\\b&-a\end{pmatrix}. $$ We have $$U(a,b)\cdot U(a,-b)=(a^2+b^2)I_2$$ whence $$ U(a,b)^{-1}=U(a,-b)/(a^2+b^2).$$ Set $$ U=\{U(a,b)\},\quad V=\{V(a,b)\},\quad v=V(1,0)={\rm diag}(1,-1).$$ Then $U$ is a subgroup of $\GO_2$, and $\GO_2=U\cup vU$. The group $\GO_2$ is not abelian. Indeed, $$ vU(a,b)v^{-1}U(a,b)^{-1}=U(a,-b)\cdot U(a,b)^{-1}=U(a,-b)^2/(a^2+b^2).$$ We obtain that the commutator subgroup $$(\GO_2,\GO_2)=U_1:=\{U(a,b)\mid a^2+b^2=1\}.$$ It follows that ${\sf X}^*(U)\cong {\Bbb Z}$ with generator $\omega$ given by $\omega(U(a,b))=a^2+b^2$. Then $\omega=d|_U=c|U$ and $\ker\omega=U_1$. We have an isomorphism $$\omega_*\colon U/U_1\to {\Bbb G}_m,\quad U(a,b)\cdot U_1\mapsto a^2+b^2.$$ The map $$ U\times\{1,v\}\to \GO_2,\quad (U(a,b), 1)\mapsto U(a,b), \ \, (U(a,b),v)\mapsto vU(a,b)=V(a,b)$$ induces an isomorphism $$U/U_1\times \{1,v\}\overset\sim\longrightarrow \GO_2/U_1.$$ Thus $\GO_2/(\GO_2,\GO_2)\cong {\Bbb G}_m\times {\Bbb Z}/2{\Bbb Z}$, and ${\sf X}^*(\GO_2)\cong {\Bbb Z}\oplus{\Bbb Z}/2{\Bbb Z}$. We have \begin{align*}&d(U(a,b))=c(U(a,b))=a^2+b^2,\\ &c(V(a,b))=a^2+b^2,\ \text{ but }\ d(V(a,b))=-(a^2+b^2). \end{align*} Thus the character $\zeta:=d/c$ of $\GO_2$ takes the value 1 on $U$ and the value $-1$ on $V$. Clearly, $\zeta\neq 1$, but $\zeta^2=1$. Thus the character $\zeta$ is of order 2. Clearly, ${\sf X}^*(\GO_2)\cong {\Bbb Z}\oplus {\Bbb Z}/2{\Bbb Z}$ with generator $d$ of infinite order and generator $\zeta$ of order 2.
Now we assume that $n>2$. We compute $\operatorname{coker}[{\sf X}^*(G)\to{\sf X}^*(F)]$. We have ${\sf X}^*(G)\cong {\Bbb Z}$ with generator $d_n$, and $${\sf X}^*(F)={\sf X}^*(\GL_{n-k})\oplus{\sf X}^*(\GO_2)\cong {\Bbb Z}\oplus{\Bbb Z}\oplus{\Bbb Z}/2{\Bbb Z}$$ with generators $d_{n-k}$, $d=d_2$, and $\zeta$. The restriction map ${\sf X}^*(G)\to{\sf X}^*(F)$ sends $d_n$ to $(d_{n-k},d_2,0)$. It follows that $\operatorname{coker}[{\sf X}^*(G)\to{\sf X}^*(F)]\simeq {\Bbb Z}\oplus{\Bbb Z}/2{\Bbb Z}$ with generator $[d_2]$ of infinite order and generator $[\zeta]$ of order 2. Thus $\Pic Y\simeq {\Bbb Z}\oplus{\Bbb Z}/2{\Bbb Z}$.
I construct explicitly a ${\Bbb G}_m$-torsor over $Y$ of order 2 in the Picard group. We consider the homomorphism $$\zeta_*\colon F\to G\times{\Bbb G}_m,\quad f\mapsto (f,\zeta(f))\,\text{ for }f\in F,$$ where we extend the character $\zeta$ of $\GO_2$ to $F$ by $$ \zeta\begin{pmatrix} A&B\\0&D \end{pmatrix} :=\zeta(A). $$ We consider the quotient $$ Y_\zeta=(G\times{\Bbb G}_m)/\zeta_*(F)$$ and the projection map $$\pi\colon Y_\zeta\to Y,\quad (g,z)\cdot \zeta_*(F)\mapsto g\cdot F.$$ Then the class in $\Pic Y$ of the ${\Bbb G}_m$-torsor $(Y_\zeta,\pi)$ is of order 2.