Humanizer 12B on 8, 16 or 32 GB: Which GGUF File to Download

A decision most people face with a local model

Imagine you have a laptop with 16 GB of memory and you want to try a local rewriter. You open the model page and find not one download but six, from 3.9 GB to 23.8 GB, with names like Q6_K and IQ2_XS-QAT. This is a hypothetical, but it is the situation almost everyone is in when they first meet a quantized model. The humanizer card is better than most at helping, because it gives each file a size, a measured peak memory, a fidelity score and a fact-check count, and says plainly who each file is for. This piece reads that table and explains what its columns mean. For the model itself, and how far to trust its headline claims, see what the humanizer numbers actually say.

The table

Top-1 agreement is how often the file’s most likely next token matches the full bf16 model, averaged over English and Chinese. Flagged is the number of English rewrites, out of 420, that a strict LLM judge found a factual problem in. Peak memory is the authors’ measurement with llama-server on Metal on an M5 Max, using the app’s settings, while rewriting. All figures are the author’s.

Why the small files are not just squeezed copies

The card draws a distinction that explains why the middle and bottom of the table behave better than you might expect. It says most GGUF files come from a single llama-quantize pass that rounds every weight to the target type. The 2-bit, Q3 and Q4_K_M files go further. For the 2-bit and Q3 files, the author measured how much each weight tensor hurts the model when squeezed and gave the sensitive tensors more bits and the robust ones fewer. All three were then trained layer by layer to reproduce the full model, which is quantization-aware training, and distilled from the full bf16 model on English and Chinese rewriting data. Q6_K and Q8_0 are standard builds, because at those sizes the card says there is almost nothing left to recover.

The effect, per the card, is measurable. For the 2-bit file, a standard build of the same size picks the same next token as bf16 70 percent of the time; the extra steps raise that to 81 percent, then 85, then 87. The details, including KL divergence and per-file fact checks, are in a document on quantization the card links; we did not verify them.

The result is still an ordinary GGUF file that any recent llama.cpp can run, which matters if you do not want to depend on the author’s app. The card also notes the files have their own repository, so existing download commands keep working.

What to pick for your memory

The card’s own recommendations are specific, and they line up with the table. With 32 GB or more, choose Q8_0, which the card calls recommended and best quality. With 16 GB, choose Q6_K, which the card says has no measurable loss. Around 14 GB, or when disk is tight, Q4_K_M, with slight loss and quantization-aware training. With 12 GB, Q3-QAT, which has small loss and a few more fact slips in English. At 8 GB, the smallest file, IQ2_XS-QAT. The app does the matching for you: on first run it suggests one of five sizes for your memory, down to 8 GB machines, downloads it once, and then works offline.

Two cautions belong beside the list. First, the memory column is a Mac measurement that leaves room for the system; the card says the app keeps about 4 GB free. On a Windows or Linux machine with a discrete GPU, what matters is video memory, and the only reliable answer is to test. Second, two rows are estimates, not measurements: Q6_K and bf16 peak memory. The Q6_K estimate of about 11 GB is the number you are relying on if you choose the 16 GB option.

Reading the quality columns without fooling yourself

The flagged counts do not fall in a straight line, and the card says why. bf16 has 52 flagged and Q8_0 has 44, which the card describes as within noise of each other. Q8_0, Q6_K and Q4_K_M are within noise of bf16; Q3 and 2-bit make a few more slips. The honest summary is that the top three files are hard to tell apart on facts, and the bottom two give up something you can see in the count, from 64 to 70 out of 420.

There is a twist in the detector column that the card is careful about. On a separate measurement, the same 60 English drafts through Originality.ai at its strictest setting, Q8_0 had 7 flagged out of 60, Q3-QAT had 4, the 2-bit file had 0 and bf16 had 4. The smallest file scored best, which the card labels as the lowest detector score alongside a few more fact slips. It is a reminder that detector scores and factual fidelity can pull in opposite directions, and that the card itself says detectors change and this is one measurement on one date. If your work involves numbers and names, the fact column is the one to weigh.

File size is also not the whole memory bill. The peak memory column is higher than the file size in every row, because the runtime needs room for context and buffers on top of the weights. The card recommends a context of 8192 tokens, covering the instruction, the draft and the rewrite together, and says to split long documents at paragraph breaks. A good habit with any local model is to treat the file size as a floor, not a budget, and leave headroom.

Other ways to get it running faster

The card documents several routes beyond the app. llama.cpp is the recommended one, with a download command, a server command and a short Python client. For Apple silicon there is an MLX path that converts the weights to 8-bit locally. For Ollama the card warns that Ollama ignores the chat template stored in the file and would apply its own, which breaks this model, so it supplies a Modelfile with the right template. For a command-line workflow with long documents, a companion tool rewrites prose piece by piece, keeps headings, code, tables and links, and flags any piece where a number went missing.

A community project is worth a mention, with the card’s own caveat that we have not benchmarked it. A contributor trained a speculative-decoding draft model for humanizer, reporting about 1.5 times the speed on an M5 Pro with MLX, 34.5 tokens per second against 22.8, and output that was token-for-token identical to running humanizer alone in their test. It runs alongside the main model and needs the MLX setup described in its own model card.

Our take

Take the biggest file your machine holds with room to spare. By the author’s own measure, dropping from Q8_0 to Q6_K costs 12 extra flagged rewrites out of 420, dropping to Q4_K_M costs 14, and dropping to 2-bit costs 26. For work with numbers and names, that gap is the thing to weigh, not the file size. If you are somewhere in the middle, the 16 GB recommendation of Q6_K looks like the sweet spot on this table: close to bf16 on fidelity and fact checks, at less than half of bf16’s size.

Keep reading on Front

Every post, newest first, is on the Front blog index.

• Humanizer 12B: what the claims say: which claims about the Humanizer 12B local model hold up against its sources.
• Ling 3.1 Flash: what it takes to run: the hardware needed to run Ling 3.1 Flash.
• DeepSeek V4.1 Flash 1017 timeline: DeepSeek’s V4.1 Flash release, its Oct 17 update, prices and parameter counts.
• Meta Muse Spark Open timeline: the dated timeline of the Meta Muse Spark Open release.
• Reflection Beam 501B timeline: the dated timeline of the Reflection Beam 501B weights release.

FAQ

Sources

• jialinyyzz/humanizer model card
• jialinyyzz/humanizer-GGUF