There already exist great tutorials how to setup IPSec for IPv6 using racoon for Internet Key Exchange (IKE) like this one or this one.
But for the usage of IPSec inside one subnet, they miss one point: Inside an IPv6-subnet clients depend on the Neighbor Discovery Protocol (NDP) for receiving Link-Layer-addresses. If NDP isn’t working, clients won’t receive the Link-Layer-address of the target host und won’t therefore be able to send packets to it.
Long story short: If you want to use IPSec for IPv6 in the same subnet, you’ll have to add an exception to enable unencrypted ICMPv6-packets. That problem doesn’t occur with IPv4, because the Address Resolution Protocol (ARP), which is used there for the same purpose, doesn’t rely on IP in contrast to NDP.
I had to search a while to figure that out and finally found the solution there: http://www.oss.sgi.com/archives/netdev/2004-02/msg00684.html.