Make sure we only use the invisible chips optimization if we are showing the "more" chip

In the case of calendar, we never shrink the field, so this optimization
can't be used

Change-Id: I30e3312b8e686be5143f96d3d2820e023158957d