may have nerd sniped myself into building a markov chain text generator from scratch with no libraries and have not slept since i last said it was 3 am
so far looking good. a fair bit of "repeating the source verbatim" but - i'm still not entirely sure on how to best tinker with the random weights to get it to be a bit more random without getting completely incomprehensible - i have not written any checks for that cuts off and retries generation, which the python library i used to use has - my corpus is quite small (i dont Post nearly enough i thibk)
i tihnk the core challenges of markov chain text generation are: 1 - keeping cruft out of your very limited context window (i can only go up to like 3 tokens before it starts commonly repeating me verbatim) 2 - managing how precise you want the context to be. in a lot of cases you probably don't actually want every token to be on there, as some of them don't really change meaning all that much and result in overfitting to exact phrasing which then ends up as repeating the corpus
this is with a fresh unprocessed db export from this instance as opposed to the previous ones which were from my bluesky pds exported using the old kopper/markov code (which does it's own processing alongside my new tokenization)
i have a horrible system in place where tokens get case-folded in the context window, and during text generation all probabilities for tokens that case-fold the same way (internally linked as "variations" on a "canonical" token) get summed up to the token that had the highest probability
also unlike the previous one i dropped the nlp library. thinking about it a bit too much i'm not sure how much benefit it really provided and just burnt cpu doing nothing
decided to throw the code out as-is to git.gay/kopper/markov if anyone else wants to give it a spin or just read through it. i still want to build a full fledged fedi and maybe bsky bot around it but that can be for later
