I'm not going to quote any reliable source here, so let that serve as my disclaimer...
Anyway, because I would say:
"Oliver and I (as opposed to "Oliver and me...") are thinking of going on vacation
I would definitely consider it OK to say:
"Oliver and we are thinking of going on vacation together."
Nevertheless, the latter sentence sounds rather odd to me, so I suggest changing that to something (more common/normal) like:
"We are thinking about going on vacation with Oliver." or "Oliver is thinking about going on vacation with us."