It’s a little tsu (tsu being the sound if it’s not small – compare to ), basically a glotteral stop (or if you want to get really technical, it is a geminate consonant)
It has no significance on its own.
In Japanese, it denotes a small delay between two sounds. For instance, if you say the word ‘rock’ (or how I say it anyway) you will find a pause between ‘ro’ and ‘ck’ (if you say it without the pause, it will sound more like ‘rawk’). You probably won’t hear it since, in English, we don’t really require it to distinguish between words that sound similar, therefore we don’t mark it, and thus don’t hear it unless we are educated to detect it.
It may mean the difference between two terms in Japanese, for example:
Kita, I arrived.
I sliced it, kitta.
They have a little difference in sound, with the latter having a short gap (so to speak) between ki and ta.
I disagree with Bellie that it shortens the vowel sound much at all (since you may speak these things as slowly as you like), but rather that it lengthens the interval between voicing one sound and saying the next. Or a more abrupt halt