Using a DarkPAN with Carton

The Problem

We work on various project for different customers. Sometimes we end up with some code that we'd like to reuse in several projects (let's call this "private libs"), but cannot release on CPAN (mostly because it's very specific to some internal problem and I'd have a bad feeling wasting community resources). We are using Carton to manage each project's dependencies, which works very well for stuff that's available on CPAN. But carton does not work too well for modules "released" via a DarkPAN0.

Workarounds

Some time ago I hacked together lib::projectroot to work around this and some other issues. It works, but it's not very elegant.

A bit later we switched to using one monorepo1 per project. This works fine for all code that's only needed inside the monorepo. But as I mentioned earlier, we have different customers, and some code we want to reuse between monorepos. Of course we could use a proper monorepo for all our code, but this causes some other problems (for example everyone now has access to all the code, even some subcontractor who only works on one small project).

Anyway, both workarounds make it rather hard to properly install the prereqs of our private libs.

Another failed experiment was to specify an absolute link to the tarball in cpanfile:

requires 'http://darkpan.internal/Foo-Bar-1.42.tar.gz';

cpanm (which is used by Carton) can install distributions this way, but Carton fails because it seems to try to load a module named "http://darkpan.internal/Foo-Bar-1.42.tar.gz" to see if the installation worked. So while it actually installs the dependency, the install process fails, which aborts the deployment.

What we really want

  • We want to pack up a private lib into a CPAN-Style tarball and upload it somewhere.
  • Then we'd like to specify a dependency on the private lib in a cpanfile as if it was a proper CPAN distribution, mixing it with regular CPAN prereqs.
  • carton install should fetch CPAN dists from CPAN, and our private libs from our DarkPAN

Prior Art

Of course we are not the first people who have this problem. One very good solution is Pinto, which allows you to set up curated stacks of CPAN and DarkPAN distributions. But I find it does not work well together with Carton (or rather, cpanfile). I like to let (for example) Dist::Zilla::Plugin::CPANFile generate a cpanfile for me, which I can later edit per hand to pin various deps to the correct version, and use carton install --deployment on some other machine to install the same environment. I have the feeling that Pinto is a too big solution for my problem, so I never properly learned it. I'm quite certain that Pinto can do all I need, but then we already have Carton embedded in our infrastructure, so a solution using Carton seemed to be less of an hassle.

Last year, Matt introduced opan which looked interesting, but again I could not get it to work after playing around with it for a bit.

Then there are various tools that let you generate your own DarkPAN, for example CPAN::Mini::Inject and OrePAn2. The latter will be of use in a bit.

The proper solution

After sifting through various search results, blog posts and source code, we finally found what seems to solve our problem: $ENV{PERL_CARTON_MIRROR}. This is a undocumented feature (or bug) of Carton, which might be worked into something proper, but probably not, as Carton seems to not longer be developed.

So we can now do PERL_CARTON_MIRROR=http://darkpan.internal carton install and Carton will install from our DarkPAN and from the default CPAN mirror. While the fact that even though I specify one mirror, Carton still uses another (hardcoded) one as a fallback seems a bit fishy, in this case it is exactly what we want.

In summary, our setup now is:

  • Private libs are developed just like small CPAN modules.
  • When we push a private lib's master branch to our gitlab repo, our CI pipeline starts to build a new tarball (using Dist::Zilla)
  • The tarball is copied to our DarkPAN server
  • A gitlab CI worker uses orepan2-inject and orepan2-indexer to add properly add the new tarball to our DarkPAN
  • Apps can list private libs as regular dependencies in their cpanfile
  • While deploying an app, we use PERL_CARTON_MIRROR=http://darkpan.internal carton install to install the prereqs, which fetches CPAN stuff from CPAN and private libs from our DarkPAN

This seems to work quite well, but if you have any improvement ideas etc, I'm happy to hear about them!

Thanks to Farhad from Spherical Elephant for helping to find PERL_CARTON_MIRROR deep in the Carton guts, and for setting up all the gitlab CI magic!

Footnotes

0 A DarkPAN is something that looks like CPAN, but is not public. To look like CPAN, you basically need a correct 02packages.details.txt.gz, which maps a package name like Foo::Bar::Baz to a distribution tarball like D/DU/DUMMY/Foo-Bar-1.42.tar.gz; and a directory tree of authors, for example authors/id/D/DU/DUMMY.

1 A monorepo is one big repo containing all your various libs, apps, tools etc in one big repo. This has some advantages, especially when you're using microservices and thus have a lot of small apps.