Merging multiple git repos with their commit history

Advent of Code has started again and I'm again publishing my solutions to github (and probably also here). In the last two years I created on git repo for each year, but this year I changed my mind and want to have on repo containing a dir for each year.

Now I could just copy all the files into the new repo, but that would lose the commit history. Which is unacceptable!

The status quo ante

.
├── advent2019
│   ├── 01_1.pl
│   └── 01_2.pl
└── advent2020
    ├── 01_1.pl
    └── 01_2.pl

What I want

.
└─── advent_of_code
    ├── 2019
    │   ├── 01_1.pl
    │   └── 01_2.pl
    └── 2020
        ├── 01_1.pl
        └── 01_2.pl

I know that git allows you to rewrite history and have once munged a repo using git filter-branch (to completely delete some files and commits). I of course forgot the details, but after a little bit of searching and trial and error I got it working!

First, prepare the old repo

To prevent merge conflicts later, I first move the code in the old repo from the root dir into a new dir, eg 2019

So this:

.
└─── advent2019
    ├── 01_1.pl
    └── 01_2.pl

Should look like this:

.
└─── advent2019
    └─── 2019
        ├── 01_1.pl
        └── 01_2.pl

I found this gist very helpful, and adapted it to my needs:

cd advent2019
mkdir 2019
git filter-branch --tree-filter 'mkdir -p /tmp/ad; mv * /tmp/ad; mkdir 2019; mv /tmp/ad/* 2019/' --tag-name-filter cat --prune-empty -- --all

This uses git filter-branch to move the old code to a temp dir (mv * /tmp/ad) and then move it back to the new location (mv /tmp/ad/* 2019/), using some git magic to keep the history.

While I was at it, I also removed some of the AdventOfCode input files which I had commited (but which we should not commit):

git filter-branch -f --tree-filter 'rm -f 2019/*.data' HEAD

Second, import the repo

Now I can merge the old repo into my new unified repo. This time this StackOverflow comment pointed me in the right direction. The basic idea is to add the old repo as a new remote, fetch the commits, and then merge them using --allow-unrelated-histories

cd advent_of_code
git remote add old19 ../advent2019/
git fetch old19
git merge --allow-unrelated-histories old19/master
git remote remove old19

And then push, and we're done and have a nice unified repo:

~/perl/advent_of_code$ tree
.
├── 2019
│   ├── 01_1.pl
│   ├── 01_2.pl
├── 2020
│   ├── 01_1.pl
│   ├── 01_2.pl
│   ├── 01_2_golf.pl
├── 2021
│   ├── 01_1.pl
│   ├── 01_1_golf.pl
│   └── 01_2.pl
└── README.md

And the history was preserved:

~/perl/advent_of_code$ git log 2019/24_1.pl
commit dbaf0bc1df645bda61ec5cf7e623d478a179947e
Author: Thomas Klausner <domm@plix.at>
Date:   Fri Dec 27 10:10:10 2019 +0100

    no need to keep a map around, the rating is unique per map

Win!

Next steps: Fix all links pointing to the old repos to point to the new location, and maybe archive / delete the github repos (and/or have them redirect to the new unified repo)