After looking at profiling output I saw that a not so small of time spent was in libarchive trying to know if archive is zip compressed... so I've enabled the three compression formats supported by xbps-src.
xbps-repo find-files