An affirmative to the "conjecture" above is fully recorded in this work:
https://arxiv.org/abs/2003.05806
In Remark 1.2, for example, we comment how Gepner-Haugseng's results imply that k-linear A-infinity categories are precisely k-chain-complex-enriched infinity-categories.
This passes through the infinity-categorical equivalence between k-linear dg-categories and k-linear A-infinity categories. The proof is completely in line with Rune's comments--in fact, it was based on a discussion I had with Rune back before COVID.
By the way, if you define a k-linear infinity-category to be an infinity-category enriched over k-chain-complexes, you're fine. (Re: Denis's comment.) But some people define k-linear infinity-categories as those with an action of the infinity-category of k-linear chain complexes; then you're not fine. Not all A-infinity-categories have (even finite) colimits, for example. I think this is what Yonatan points out in the comments.
So, to address OP's original question: No, k-linear A-infinity-categories are not a full subcategory of infinity-categories.