Delhi could lose the Commonwealth Games in 2010 if organisers have to move the athletes' village from its current site.