As mentioned in the comments, the book above discusses only the notation for bilinear forms. For n-ary forms Cayley gives a more explicit description of his notation here page 413: $$(a,b,c,f,g,h,i,j,k,l)(X,Y,Z)^3$$ with the right symbol instead, stands for $$ax^3+by^3+cz^3+3(fy^2z+gz^2x+hx^2y+iyz^2+jzx^2+kxy^2)+6lxyz$$