Why wasn't Sri Lanka, or Ceylon then, in the British Raj of India?
Mainland India and Sri Lanka are pretty close, or was it down to the ethnic groups and religions?

Google up "British Raj of India".  Wikipedia has an extensive and rather complex article on this. Near the top of the article, under 'Geographical extent', is this:

"Among other countries in the region, Ceylon (now Sri Lanka), was ceded to the United Kingdom in 1802 under the Treaty of Amiens. Ceylon was a British Crown Colony, but not part of British India."

There is a link to 'British Crown Colony' which sheds some light on this.  However, exactly why may not be fully answered in wikipedia, or if it is, might take some digging out, but if you're interested in this, you'll see how that area became what it is today.