Can one use the real lie algebra so(3) to get knot polynomials? If so, do they have a skein relation (I presume they would, if they come from Rmatrices in some standard way. If so, is the Rmatrix written down anywhere?) I've only ever seen complex Lie algebras used to get knot polynomials, but I'm not sure if this is for some fundamental reason, or just that no one has bothered to investigate the real ones.

I don't see how taking a real form would change your knot invariants at all. Whatever it meant would be a number which would then not change when you base extend to the complex numbers. From the quantum group perspective U_q(so_3) and U_q(su_2) mean the same thing (as far as I understand it), but for so_3 the natural representation is 3dimensional, whereas for su_2 the natural representation is 2dimensional. So the Rmatrix you want is just the Rmatrix for the three dimensional representation of U_q(su_2). I'm not sure if that's been written down somewhere that's easy to find, but it's a good exercise to work out yourself. From the diagrammatic perspective there's something called the "Yamada polynomial" which gives the so_3 knot polynomial (just as the Jones polynomial gives the su_2 knot polynomial). The other thing people mean when they say SO(3) instead of SU(2) is that you only look at the category of representations of U_q(su_2) which come from deformations of representations of su_2 which lift to the lie group SO(3), that is only the representations whose highest weights are in the root lattice. The best place I know to learn about this perspective (and in particular what it means in terms of 3manifold invariants which is what Jose mentioned) are the papers of Steve Sawin, for example this one. 


Edit: As pointed out in the comment below, the question is about knot polynomials constructed out of the Rmatrix, whence my original answer is not relevant. I'll leave it here just in case someone else misunderstands the question. Of course you can. In the ChernSimons approach of Witten's you can get knot polynomials for any compact simple Lie group. (In fact, ChernSimons theory for noncompact Lie groups is still not wellunderstood despite continuing progress, reviewed in a very recent paper of Witten's.) In fact, the knot polynomial associated to $\mathfrak{so}(3)$ is the Jones polynomial. 


Check out Theorem 4 of BarNatan's paper (based on his thesis). He proves that you can obtain a weight system from an adinvariant bilinear form on a lie algebra defined over a field $F$ together with a representation. When $F=\mathbb{R}$, then Kontsevich's theorem implies that the weight system may be integrated to a knot invariant of Vassiliev type. He also proves the equivalence of these invariants with the quantum invariants. So I think that the Lie algebra only needs to be defined over $\mathbb{R}$. I don't know whether Kontsevich's theorem has been generalized to other fields. However, weight systems may be derived from other algebraic structures such as super Lie algebras (Vaintrob). 

