Day 9 of self-studying ML — entropy, cross-entropy, and logistic regression notes [D]

new set of notes up, covering entropy and information theory then working through KL divergence and cross entropy, ending with why logistic regression's loss function is called cross entropy loss in the first place

the connection to the homework problem is what made it worth posting. the assignment has you show that maximizing the likelihood of the labels is equivalent to minimizing J(w), the loss function from the logistic regression chapter. you write out the likelihood as a product across the dataset, take the log to turn it into a sum, and the negative of that log likelihood turns out to be exactly J(w). so cross entropy loss isn't just a name someone picked for it, it comes directly out of maximizing likelihood

what helped was doing the entropy notes first. once you've seen H(p,q) equals D(p, q) plus H(p), and worked through why KL divergence is basically the extra bits wasted from using the wrong distribution, the logistic regression derivation stops feeling like a formula to memorize and starts feeling like the obvious result of penalizing the model for putting probability mass in the wrong place

full notes are on my github, link's in my profile if you want the pdf

anyone want me to share how I've been structuring the whole self study path, course order and what came before this