I've actually managed to get the offset working! I just stored the last position in a variable once the finger is lifted and added it to the current position vector.
However, my problem now is that everytime the screen is touched the centre of the map kind of snaps to the finger, and then it can be dragged as expected. So my question is, how can I make the location the finger touches on contact be 0,0,0?
Thanks!