User:Kim/Prototyping 2nd year/GitPub

log

prototype status	v 0.5 (iso)
remote	https://gitlab.com/km_kt/gitpub-scripts
initial pmomm	#1. gitpub

v 0.1

utilizing the Git API.

• 

  gitpub v1 scripts

• 

  gitpub v1 printed prototype

I manged to print a proof of concept version of my personal reader repository but ran into a lot of issues:

• API needs personal token authentication (which limits it to personal use)
• using curl (to json) is limiting in formatting options (v1 is very 'crafty')
• sorting the files and directories is tricky and you prob. need to do it manually by passing numbers to the titles

v 0.2

I dont need the Gitlab APi, since most meta info is already stored in my local repository. try python script >> better to turn into a program (pip install gitpub) using uv

find the python script here here

• 

  gitpub.py (v2) script

wip ref

• https://thelinuxcode.com/git-log-command-in-git/
• Git internals are explained further in this (very compelling) chapter called Plumbing & Porcelain (referring to underground / lower level commands vs the more user friendly / surface commands like git commit etc) >> it gets very complicated very quickly
• The architecture of open source application version 2 Git by Susan Potter
  
  

where is the local information stored?

in the repository, commit messages can be found in .git/logs/refs/heads/main
they are structured like this:

0e55d65009ff9617a72b0eafc908e356a8344bb4 5567bfffaad5ab96941b2cd7ed8aa077c2df1e94 Kim Kleinert 
<mail@kimkleinert.com> 1759126065 +0200 commit: r w o t synopsis

while when typing git log the output is formatted as the following: (git log documentation)

commit 5567bfffaad5ab96941b2cd7ed8aa077c2df1e94 (HEAD -> main, origin/main)
Author: Kim Kleinert <mail@kimkleinert.com>
Date:   Mon Sep 29 08:07:45 2025 +0200
   r w o t synopsis

viewing the same commit in the gitlab interface reveals that 5567bfffaad5ab96941b2cd7ed8aa077c2df1e94 is the SHA while 0e55d65009ff9617a72b0eafc908e356a8344bb4 is indicated as the parent (which usually indicates the previous commit). that still leaves this string 1759126065 inside the git log file view unknown.


how can I get it?

>> use the git log cmd
to get log from specific file: $ git log file/path/name.md
options:

• view diff by adding -p
• --oneline displays only hash and message in one line
• --pretty=format:"..." documentation flag to have more control over formatting output git log --pretty=format:"%s" queue.md(this only print commit message), --pretty=format:"%an: %s"( author name and commit message) and adding %as for date
• --stat to get only the amount of changes not the full diffs

>> use python library$ pip install GitPython? this "provides a higher-level API for interacting with Git repositories" and is recommended for more complex interactions with git
or start with subprocess (more general for bash) or os >> why am i using python again?

in between questions and steps

• display filenames in final pdf (headers?)
• make it editable collectively (display merging branches, user names)
• file sorting before or with pandoc
• expand file formats
  • make it terminal interactive: let user choose file formats (not only .md)
  • in bash this could work to get the file contents into .md cat style.css > out.md (does it preserve formatting?)
    • and the python equivalent
• expand styling: how do the commits interfere with/ relate to the main printed text?
  • look into pandoc templates

v 0.3

gitpub2.py

• 

  trying to figure out gitpub2.py on 3 different paper media and 2 screens

• 

  gitpub v0.3 graph

changes made:

• accepts user input to choose file formats (only .html and .md work)
• tested on different repository (make copy, add .py script, template.html and css file)
• runs without virtual environment (but needs pandoc, weasyprint and pdfbook2 preinstalled)
• first builtin conditionals (prevent converting hidden files and dirs)
• metadata use in v 0.3:
  • markdown to single html then glob html
  • add metadata (title) in the pandoc cmd to html
• use of 2 custom html templates, applied for each text and -cm files
• running headers (from filetitles) using print css and custom template class
• book title from cwd name

changes to make for mock assessment:

• move new templates and script to writing-from-the-middle-copy directory
• styling:
  • adjust margins for printing
  • format / position metadata (to the side? larger than main text? possibilities to keep its formatting / line breaks -- adjust in .py script?)
  • cover with repository title (and url would be cool)
• print
• bind

practical info

markdown metadata

• pandoc accepts metadata in yaml: (can be added as separate yaml file or inline .md document)

---
title: document
author: kim
---

• it is also possible to add a title block

% title
% author(s) (separated by semicolons)
% date

without editing .md files directly, I can pass metadata to the file by using this markdown command pandoc -s --template ./template.html --metadata title="$title" "$oldname" -o "$newname" found here

templates

• pandoc lets you add [1] for standalone documents (-s)
• github repo with all pandoc default templates
• to test which template you are currently using pandoc -D html (exchange html with respective format converting to)

formatting

• in the script i could run different templates on -cm.md files than original .md files for styling differently, experiment with merging
• doing it really properly: pass all the git metadata to markdown file as yaml metadata > that way I can style it well with a custom template

v 0.4

• 

changes

• pandoc is now only running once: converting .md or .html (so far) to individual .html files
  • patching all .html files together is now done in python (double html conversion in pandoc is causing misformatting)
• gitlog output / metadata for each file is written to .yaml file (instead of -cm.md)
• added citation rendering option (--citeproc) *(['--cite'] if cite else []) inside of subprocess.run this inserts argument only if variable is true
• added custom stylesheet option
• references are rendered into separate file. How does it work? see reference rendering section

errors

• can find a way to format git log --stat output into .yaml (its removed for now) -- might need an extra script?
• TOC is currently unavailable (was previously rendered with pandoc) necessary?

general considerations

remove weasyprint / pdfbook2? they seem a bit unnecessary as dependency when you can print / export form the browser directly...

reference rendering

I documented the set up I am using for thesis writing here: User:Kim/Stations, Skills, Resources/VS Codium - Zotero - BibTex. The basis is a standard .bib file in my repository, which pandoc with the --citeproc argument, uses to 1. render in text citations and 2. output a bibliography at the end of every file.

1.is desired, while 2. should be changed into a global bibliography, separate document, for all files. I used this solution and adjusted it to my python script (it's a bit crafty):

when the user input citation = true

1.--citeproc option is added to pandoc command

2.create a variable that contains yaml metadata needed for .bib rendering:

bibliography: ./references.bib
supress-bibliography: true

In this I define the .bib file for citation rendering and prevent the creation of a file specific bibliography. this will be added in the git_to_file function where gitlog output is written to .yaml

3. create a reference.md file with the following content:

---
nocite: '@*'
bibliography: references.bib
---
# References

.md file with yaml header, specifying nocite with wildcard renders the full .bib file content, bibliography specifies (as above) defines our .bib file path and the # References creates the bibliography output container

observations

1. (pandoc) when two metadata input sources are defined e.g. external .yaml file and inline yaml header, the yaml header is used over the external file.
2. for securely outputting validated yaml, the git commit is now formatted like this (in --pretty) commit: >-%n %s%n. this makes sure characters like colons :, which yaml syntax uses, dont cause errors

pending

• currently all sources are rendered (from the .bib file) into bibliogrpahy and not only the ones in actual use
• (back)links in the bibliography to the citation are currently breaking
• make sure reference file is appended at the end of output html (sorting matter)
• validation, if cite user input is true, make sure .bib file exists

next

1. file sorting
2. visible branching (git log --graph)
3. see errors

v 0.5

in progress

changes

• errors from 0.4 fixed (see below pydriller & beautifulsoup)
• v 0.5 runs with venv to manage libraries (see Python Notes)

PyDriller

PyDriller is a python library to extract information from git repositories, especially focusing on commit data output.

1. to get file commit information, especially those not inlcuded in git log format pretty (deletions, insertions, branches)
2. pydriller, other than GitPython, goes deeper (e.g. automatically contains options --follow --all, which to get all the commits, also after file renaming)

beautifulsoup

Beautifulsoup is a python library for parsing html to manage html files more precisely

1. to patch individual html documents from ./html into one
   1. !DOCTYPE declaration missing
   2. meta name viewport missing (causes error)
2. to iterate over headers and create toc

pending

1. split main script into smaller ones than can be run from main
2. file sorting
3. visible branching (?) git log graph
4. optional toc (user input)
5. from citation (basic) v 0.4:
   1. validation for .bib file if citation user input is true
   2. reference file always rendered / appended last
   3. currently all sources from .bib are rendered (possible to only include those in use?) otherwise rm manually

v 0.5 iso

separating the script (and rendering files) from the repository it is applied to. (Although not the main goal) this step makes it easier for the gitpub to be applied to and tested on existing repositories.

• 

1. the gitpub-iso script can be run from an external git repository (script repo).
2. the script asks for a local, absolute file path to the repository it should applied to (project repo).
3. it will then fetch the project repos metadata and render the output in the script repo.

notes:

the script repository does not have to be a git, but can be any directory containing the specified files and installations.

v 0.6

changes made:

1. added paged js (for browser preview and pdf conversion)
   1. added parallel flow plugin to paged js to format metadata output
2. using python yaml to ensure correctly formatted commit messages
3. final html document uses jinja template
   1. allowed to restructure html for parallel-flow plugin to work
4. adjusted user input options: (file types, cover, css and citation)
   1. citation has validation for .bib file
5. toc is rendered in python (beatifulsoup), currently has 1 level (thesis version has 2)
6. cleaned up file structure:

.
├── README.md
├── assets
│   ├── css
│   │   ├── interface.css
│   │   └── style.css
│   ├── js
│   │   ├── csstree.js
│   │   ├── paged-polyfill.js
│   │   └── parallel-flows.js
│   └── templates
│       ├── cover.html
│       ├── document.html
│       └── template.html
├── gitpub6.py
├── html
│   └── README.html
└── yaml
    └── README.yaml

pending / further:

1. make different templates for jinja html output (currently, its optimized for paged js output but could also be web...)
2. remaining: file sorting

outlook

02/02/26 tutorial with Kamo

1. for formatting git log output (other than whats included in pretty) look into a library like GitPython
   1. this tutorial provides a good example of working with GitPython and formatting log output
   2. GitPython reference for the commit object
   3. otherwise this script parses git log output to json
   4. use pydriller which seems to be made to parse commit output in python specifically (found in stackoverflow)
   5. libraries can be included into the python project using UV (dependencies)

issues with gitpython: the git log needs --follow --all in order to get all the commit messages, but apparently I can add these to repo.iter_commits(all=True, paths=file) (when iterating over commits from a file) so I always only get the last one?

1. use Flask to automatically update a website (html file) or pdf on every push (this becomes relevant if I dont want to run the script manually every time to see the output, especially when collaborating... but a question remains that in order to implement this, there has to be something on a server? while now everything runs through git - local and remote). here are some flask resources:
   1. Manetta on GitTea Flask Example
   2. xpub wiki Flask
   3. Kamos Thesis git repository and the thesis viewer rendering markdown into a static html page using flask
2. connected to flask are webhooks, that you can set up on the gittea or gitlab platform itself. they define under what parameters the website (or file) it is connected to should be updated (on commit or push or every page visit...)

interface

in the writing process, how can metadata / commit messages become part of the process instead of only being visible as and end / byproduct?

• flask to render to html (or pdf) more continuously
• share html on localhost (enter same localhost with same ip address, bc we dont have a server) (maybe needs (web) server configuration, apache / nginx)

repository connection

keep the scripts in a separate (from your project) repository that you also dump the metadata output files in. This way the working repository stays 'clean' and you can apply gitpub to diverse projects.

see: Gitpub

git blame

use git blame instead of git log to let the text unfold through its metadata (commits)