In the early days of psychiatry, disorders were split into two categories, “neurosis” (including disorders we would now call anxiety and depression) and “psychosis” (including syndromes resembling schizophrenia and mania). Patients that had characteristics of both groups were described as being “borderline”, as in they sat on the border demarcating the two categories.
From Wikipedia: “The first significant psychoanalytic work to use the term “borderline” was written by Adolf Stern in 1938. It described a group of patients suffering from what he thought to be a mild form of schizophrenia, on the borderline between neurosis and psychosis.”
There is a general trend away from this diagnostic label, as it does not tell people anything useful about their condition now that the dichotomy between neurosis and psychosis has been largely forgotten in the face of more detailed classification systems. “Emotionally unstable personality disorder” is the ICD-10 name for the same condition.