Do you connect the pogo pins before powering on the USBasp? I’m not sure about the protection of the ATtiny, but I made bad experiences with other controllers when the ground has bad contact and a positive voltage is applied first. That’s why I always connect the pogo pins first and then plug in the USBasp into the USB port.