I recommend installing a few basic packages to provide a bare-minimum build environment:
Then install cpanm:
cpan install App::cpanminus
And use that to install stuff.