Recently I am working on a new convex hull algorithm. I would like to know if there is any forum where I can submit my work.
Computational geometry has several journals (Discrete and Computational Geometry, Computational Geometry: Theory and Applications, International Journal of Computational Geometry and Applications, and Journal of Computational Geometry) and conferences (ACM Symposium on Computational Geometry, European Workshop on Computational Geometry, Fall Workshop on Computational Geometry, Canadian Conference on Computational Geometry), and its own section (cs.CG) of the arxiv.org preprint server. There is an even larger number of journals and conferences for algorithms research more generally, many of which are welcoming to computational geometry, and there are other related conferences such as the Symposium on Geometry Processing in the computer graphics sphere of influence.
As with the rest of theoretical computer science, researchers in this area tend to pay much more attention to conferences than to journals, so if you want them to notice your work then sending it to a conference instead of a journal would be a better choice. (If the details of the paper do not fit into the conference page limits, it's usually possible to make an expanded journal version of the same material after the conference, but it's usually not possible to get a paper into a conference if it is already in a journal.) Among the conferences I mentioned above, the Symposium on Computational Geometry is quite difficult to get papers into, the others much less so. The next submission deadline is the one for CCCG, in May (the conference is in August); see http://2011.cccg.ca/ for details. And if you do intend to submit to one of these conferences, be sure to read through examples of papers from previous years to get some idea of what their papers are expected to look like. (Using LaTeX instead of some other word processor is also highly recommended.)
Getting your paper onto arxiv.org requires only that it be on-topic, and doesn't pre-empt any other kind of publication, but unless you're well-known in the field, your work is much more likely to be taken seriously if you hold off on putting up a preprint until you get it accepted to a conference first. And even if you are well-known, waiting until you have some peer review before making a preprint is usually a good idea.
All that said, convex hull algorithms are a rather heavily worked-over field, and in many cases the known algorithms match or are very close to matching the worst-case lower bounds for the problem. So for your work to have an impact, you need to demonstrate why it is better than its competitors either through experimental comparisons with state-of-the-art implementations on realistic data or through rigorously proven bounds on its time complexity that are a clear improvement on the time bounds of the known algorithms; just being a new algorithm isn't going to be enough.
You could post it, or a link to it, to the Usenet newsgroup sci.math. This will get you some confidence-building peer approval, if your work is good (although it is not a substitute for publication in a journal or in conference proceedings), but the real advantage is that if your work is no good you won't have to wait months for a rejection; your work will be torn to shreds with little delay.