The singularity avoidance problem in image-based visual servoing can be formulated as a constrained optimization problem where the constraint is typically the distance from singular points. This paper develops a new approach to vision-based robot control, which avoids the robot singularities by using a new method for nonlinear programming, called biased Quasi-Newton. Contrary to classical singularity avoidance this approach does not need any a-priori knowledge of kinematics or robot model to avoid singular points or regions. Furthermore we also illustrate how to apply the bias method for simple visual obstacle avoidance. Finally to show the practical applicability of our method. We have implemented it on Barrett WAM and PUMA560 manipulators and tested both numerous real trajectories, as well as run exhaustive simulations around critical configurations using a simulation model to confirm empirically that both safe and efficient trajectory are chosen around singular regions.