Tucked away on a roof off the western corner of Russell Square is a small, but perfectly formed, Japanese garden. It's above the Brunei Gallery at SOAS, part of the University of London, and was built in 2001. A quiet space, it's enclosed on all sides (though you can peek at — and be peeked at in return — by people in an office on one side), with a little raked gravel section and wisteria hanging above benches, which will be lovely in spring.
The garden is dedicated to forgiveness, which is the kanji character carved into the base of the water basin, and it really is perfect for a spot of contemplation. The stage area is occasionally used for tea ceremonies or musical performances. Right now, it's just got some pots on it.
You can visit the garden for free any time the Brunei Gallery is open: Tuesday to Saturday 10.30am-5pm, and open until 8pm on Thursdays. The gallery sometimes closes to change exhibitions so maybe keep an eye on its Facebook page.
Other little gardens in London: