I have an optimization problem, this problem has linear constraints and nonlinear constraints. I solved the linear part by MATLAB but the nonlinear constraints I could not solve it. I downloaded Barone software to solve it but this software has a limit of constraints. Where if these constraints more than 10 constraints then I have to purchase the licenses and the license is expensive. Do you know any free program to solve my problem or any Idea?
Given that you already have MATLAB, you can do this with software available for npo extra cost. Specifically, use the BMIBNB branch and bound global optimizer included with YALMIP https://yalmip.github.io/solver/bmibnb/ , together with a local nonlinear optimizer which is called as "upper solver" to solve subproblems within the global solver. IPOPT (available under OPTI toolbox) can be used for this purpose, as can a number of other solvers available at no extrea cost or for extra cost.
Alternatively, if you don't run into computation resource limits and are willing to formulate your problem using GAMS, you can use BARON for free, even on problems exceeding the free limits, by submitting your problem to the NEOS server https://neos-server.org/neos/solvers/go:BARON/GAMS.html .
MiniZinc is a free and open-source with IDE and many solvers.