What about using a spacer with your AAA holder? You could make spacers from aluminum bushings or a piece of copper wire. I used some #4AWG copper wire and PVC tubing to make a few spacers of various sizes to use in flashlights.
Also, I don’t know if it would work, but possibly you could cut out a middle section of your 4xAAA holder and fasten the two ends back together to make a shortened version. I might try that for mine, since the charger is made for 4x cells, and I have at least that many 10180 cells that could need charging at the same time.