But it’s so good at programming if you already know how to program! Surely that’s worth burning the planet and crashing the world economy??
Actually still no
https://github.com/JustVugg/colibri
Everyone was desperate to be first because capitalism. But we are getting good models without the insane build out requirement. Which will be hilarious to leave the cunts holding the bag. Not that the planet is better for it in the end.
~1 token per second (storage bound gen4 nvme)… Some of us have places to be.
Don’t get me wrong. Its impressive that it can run at all, but honestly the usecase is exceedingly narrow. You’d have better results with a structured quantized gpu-only gemma or qwen workflow. Quality over quantity, rely on validation and a structured process: lots of cross-model review and iteration loops with spec and test driven dev. You could probably get a working alpha by the time colibri set up the environment.
Yeah I’m just beginning my local AI journey on a 5080, tried Qwen3.6 27b Q4 and was getting like 1tps because of the vram overflow. Ran it over night at it was still chewing on generating a prompt for a sub agent when I got up in the middle of the night until it simply ended in some kind of “fetch failure” lol. I think I gave it something too large to tackle, but either way 1tps is kinda garbage.
Is that quantized? 4 bit Qwen 3.6 can get 22tps on a 1060.
It’s the q4 quantization, but it requires 20+GB vram and my 5080 only has 16
What are you using to run the model? Llama.cpp will automatically split the model between your system ram and graphics card’s vram.
Qwen 3.6 is a mixture of experts model with only 3B parameters active at a time. Even without quantization your card could easily run that.
It has been some time since my initial comment so at the time I was mainly using LM studio. Qwen 3.6 a3b is the MOE and it does work well on my card, but the dense model that is more intelligent/capable is the Qwen 3.6 27b which doesn’t fit on the card and does get offloaded, but offloading cuts the speed down to like 1/tps.
I have since found a version of the 27b model that is “quantized,” for lack of a better term, differently and has to be run through TabbyAPI which gets back to 30ish tps. It can’t offload so it must fit fully on the card which keeps the speed high. Might be worth a look if you’re interested, the only downside is that with my 16gb card the context limit has to be kept pretty low ~40k if I remember correctly
Man I swear, AI is like astrology, people have strong opinions about it but haven’t done any research into what it actually is. So much more nuanced than ‘AI bad!’
AI is like astrology
Please keep going.
Thousands of years of research into a short interpretation
Please tell me more about astrology as a millenary discipline.
Astrology is a millennia-old discipline built from the systematic observation of planet and star cycles and their perceived correspondence with earthly events. It developed alongside astronomy, mathematics, calendrical science, agriculture, medicine, and political forecasting across several civilizations. You don’t have to accept its interpretive claims, but reducing it to newspaper Sun signs is a bit like reducing literature to a fortune cookie
You don’t have to accept its interpretive claims
What’s the value, then? It sounds like its results are correlations between loosely interpreted measurements, i.e. two identical sets of observations may correlate to different events, depending on subjective post hoc interpretations.
Is that what you mean when you compared this to AI? Because LLMs are stochastic parrots?
I think the comparison is that both fields are often misunderstood through oversimplified descriptions. “Astrology is just Sun signs” and “LLMs are just stochastic parrots” are both reductionist summaries that leave out a lot of the interesting discussion. Whether either ultimately succeeds at what it claims is a separate question.
As for astrology itself, I became interested through Carl Jung, who viewed astrology as a symbolic system that reflected recurring psychological patterns and wrote about it in relation to archetypes and synchronicity ( Man and his symbols .) Astrology provides a symbolic language for thinking about cycles, temperament, motivation, and meaning.
I think you misunderstood. I don’t believe calling LLMs “stochastic parrots” is accurate, but I think that dismissing them as such puts them at the same level as astrology, in the sense that both would be useless in order to predict an outcome based purely on their inputs.
In other words, I do think that LLMs have some value to them, even when issues like “hallucinations” will never be resolved.
On the other hand, I don’t think astrology is valuable at all. There hasn’t been a single accurate prediction made by astrologers that could be replicated within the same conditions at a different time. Descriptions of phenomena in astrology are also deliberately vague, so they could be “massaged” to fit the outcomes. Astrology is, at its core and necessarily, post hoc, so suggesting that it is a misunderstood discipline based on actual science is just naive.
Look, if you know a way to convert a PDF to text with less than 500GB of VRAM and 2000W of power used for twenty seconds, I’m all ears.
Runs on anything that runs Linux:
NAME pdftotext - Portable Document Format (PDF) to text converter (version 3.03) SYNOPSIS pdftotext [options] PDF-file [text-file] DESCRIPTION Pdftotext converts Portable Document Format (PDF) files to plain text. Pdftotext reads the PDF file, PDF-file, and writes a text file, text-file. If text-file is not speci‐ fied, pdftotext converts file.pdf to file.txt. If text-file is ´-', the text is sent to stdout. If PDF-file is ´-', it reads the PDF file from stdin.
