Ideally we want our cost/loss function to be convex(concave).
First we give the definition of a convex set: A set C in \mathbb{R}^n is said to be convex if, for all x and y in C and all t \in (0,1) , the point (1 − t)x + ty also belongs to C. Geometrically this means that every point on the line segment connecting x and y is in C as discussed below.
The convex subsets of \mathbb{R} are the intervals of \mathbb{R} . Examples of convex sets of \mathbb{R}^2 are the regular polygons (triangles, rectangles, pentagons, etc...).