Remove useless map in SwapAllocator

This patch removes a map in SwapAllocator because there is no need for it.

Change-Id: Ifbc641d0d919eaaae7994836dfe423883046c61f
Signed-off-by: Jean-Philippe Halimi <jean-philippe.halimi@intel.com>