Use git-lfs to Manage Large Files in My Blog

I have been a fan of org-mode for more than four years, I use org-mode to record my feelings, write notes/blogs, and organize my life. Though most of my notes and blog posts are only halfway done, I still feel it beneficial to write down something.

Most of my friends are using notion to organize their notes and to discuss ideas. I admit that notion is better for teamwork and might have a large user base. However, I prefer a private informal space (like this blog) where I can write random text whenever I want, and I should have complete control of this space (all contents should have offline backups and version control).

Fortunately, org-mode has all of the functionality I want: a powerful markup language with code and formula rendering support, jupyter-like interactive code blocks, decent to-do list (though I don't always finish things on time), and tools for exporting org files to html/latex so I that I can share some of my posts with others. Compared to markdown and static site generators, org-mode is not only a markup language but also an ecosystem that has tight integration with emacs. Honestly speaking, I don't use emacs to write code anymore¹ but it's still highly customizable and convenient for writing articles.

My blog is basically an org-mode repository where I use git for version control, I bought a VPS as a git server, and whenever there is a commit, the server would execute emacs' org-export function to generate static HTML files, this could be implemented with a post-receive hook on git server easily.

Git was originally proposed for version control of codebase. However, I use figures to explain ideas in my blog post sometimes, and I also organize my photos with org-mode. It's well known that git is not good at large binary files ², one alternative is to use a standalone service to host images such as Amazon S3, considering I already own a VPS I feel it unnecessary to rent another service.

git-lfs was proposed to aid git with large file storage. The idea of git-lfs is simple, in their own words:

Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like GitHub.com or GitHub Enterprise.

User can specify some file types as binary files, text files and binary files are treated differently: binary files are stored in a standalone large file storage, while their pointers are treated as ordinary text. There is an external smudge phase where pointers are rewritten to their contents when we perform git lfs pull or something similar.

git-lfs team maintains an official release of git-lfs client, but they do not provide an up-to-date official git-lfs server implementation: instead, they have designed a set of standard APIs so that different git-lfs service providers can implement these APIs in their own ways. Github and Atlassian repos has lfs support but Github only provides 2GB free lfs budget, however, my blog repository has grown beyond this size, and personally, I prefer using git-lfs on a self-hosted server instead of Github. Here is a list of unofficial implementations of git-lfs server but most of them are out-of-maintenance or personal projects just for fun. For sake of safety, I choose gitea because it has an active open-source community (I suppose gitlab's user experience is similar).

Error downloading object: ???.jpg (hash): Smudge error: Error downloading ???.jpg (hash): error transferring "hash": [0] remote missing object hash 

#!/usr/bin/env bash
export COMMIT_ID=$(git rev-parse --short HEAD)
export WORK_TREE=/var/www/${DOMAIN}
export LOG_PATH=/var/log/${DOMAIN}/${COMMIT_ID}
export GIT_DIR=$(git rev-parse --git-dir)
mkdir -p ${LOG_PATH}

echo "link gitea lfs objects: starts"
if [ ! -d ${GIT_DIR}/lfs/objects ]; then
    echo "${GIT_DIR}/lfs/objcets do not exist, create directory..." > ${LOG_PATH}/LOG
    mkdir -p ${GIT_DIR}/lfs/objects 2>${LOG_PATH}/ERR
else
    echo "${GIT_DIR}/lfs/objects already exists" > ${LOG_PATH}/LOG
fi

for oid in $(git lfs ls-files -l | cut -c -64); do
    prefix_0=${oid:0:2}
    prefix_1=${oid:2:2}
    suffix=${oid:4:60}
    if [ ! -d ${GIT_DIR}/lfs/objects/$prefix_0/$prefix_1 ]; then
        echo "${GIT_DIR}/lfs/objects/$prefix_0/$prefix_1 do not exist, create directory..." > ${LOG_PATH}/LOG
        mkdir -p ${GIT_DIR}/lfs/objects/$prefix_0/$prefix_1 2>${LOG_PATH}/ERR
    else
        echo "${GIT_DIR}/lfs/objects/$prefix_0/$prefix_1 already exists" > ${LOG_PATH}/LOG
    fi
    if [ ! -f "${GIT_DIR}/lfs/objects/$prefix_0/$prefix_1/$prefix_0$prefix_1$suffix" ]; then
        echo "${GIT_DIR}/lfs/objects/$prefix_0/$prefix_1/$prefix_0$prefix_1$suffix do not exist, link from gitea lfs..." > ${LOG_PATH}/LOG
        ln -s ${GITEA_LFS_PATH}/$prefix_0/$prefix_1/$suffix ${GIT_DIR}/lfs/objects/$prefix_0/$prefix_1/$prefix_0$prefix_1$suffix 2>${LOG_PATH}/ERR
    else
        echo "${GIT_DIR}/lfs/objects/$prefix_0/$prefix_1/$prefix_0$prefix_1$suffix already exists" > ${LOG_PATH}/LOG
    fi
done
echo "link gitea lfs objects: finished"

# deploy blog
echo "git checkout files"
git --work-tree=${WORK_TREE} checkout -f > ${LOG_PATH}/LOG 2> ${LOG_PATH}/ERR
cd ${WORK_TREE}
echo "make html"
make all > ${LOG_PATH}/LOG 2> ${LOG_PATH}/ERR

I found it not an easy job to upgrade my blog repo to use git-lfs and modify the deployment process correspondingly. But anyways it really improves my experience, the waiting time per pull/push is reduced significantly. There is a famous article encourages people not to use git-lfs, though I agree on some of its arguments, I will stick to my decision until there is an (apparently) better choice than git-lfs.