Some of the symbols used in that time would be tricky to type in Latex, so instead of writing an explanation here, I hope it is okay to just give a reference. There are two books by Florian Cajori, "A History of Mathematical Notations: Vol. I and II", which are a good reference for this specific kind of question. Cayley's and related notations on determinants and n-ary forms can be found in vol II, starting at page 87, particularly look at page 94.
As mentioned in the comments, the book above discusses only the notation for bilinear forms. For n-ary forms Cayley gives a more explicit description of his notation here page 413: $$(a,b,c,f,g,h,i,j,k,l)(X,Y,Z)^3$$ with the right symbol instead, stands for $$ax^3+by^3+cz^3+3(fy^2z+gz^2x+hx^2y+iyz^2+jzx^2+kxy^2)+6lxyz$$