What are the key data structures which form the CPython runtime and how are they initialized at startup
A deep dive into CPython's bytecode instruction format and execution engine internals
How costly it is to call functions and builtins in your python code? Does inlining help? How have the recent CPython releases improved performance in these areas?
Ever wondered how your CPU handles two tasks at once? Discover the magic of Simultaneous Multithreading and see what’s really going on inside.
Learn everything about hash tables, collision handling, and performance optimization
A detailed code walkthrough of how CPython implements memory management, including reference counting and garbage collection
How Python Compares Floats and Ints: When Equals Isn’t Really Equal
Another Python gotcha and an investigation into its internals to understand why this happens
How Python Compares Floats and Ints: When Equals Isn’t Really Equal
Another Python gotcha and an investigation into its internals to understand why this happens
What powers the ground breaking performance of Groq's Langauge Processing Unit?
A series of videos of me taking you through the CPython source code to explain its type system design & implementation
Understand the mechanics of dynamic dispatch implementation in CPython
Understand the mechanics of dynamic dispatch implementation in CPython
Understand how objects are implemented in CPython and how CPython emulates Inheritance and Polymorphism in C using struct embedding
Understand how objects are implemented in CPython and how CPython emulates Inheritance and Polymorphism in C using struct embedding
Cracking the Code: Fast Fibonacci Computations With a Linear Algebra Twist
A primer on GPU architecture and computing
A primer on GPU architecture and computing
A primer on GPU architecture and computing
A detailed analysis of the DeepMind/Meta study: how large language models achieve unprecedented compression rates on text, image, and audio data - and the implications of these results
A detailed analysis of the DeepMind/Meta study: how large language models achieve unprecedented compression rates on text, image, and audio data - and the implications of these results
Interesting. I'm just thinking aloud to understand this.
In this case, the models are looking at a few sequence of bytes in their context and are able to predict the next byte(s) with good accuracy, which allows efficient encoding. Most of our memories are associative, i.e. we associate them with some concept/name/idea. So, do you mean, our brain uses the concept to predict a token which gets decoded in the form of a memory?
Yes. They also mention that using such large models for compression is not practical because their size thwarts any amount of data you might want to compress. But this result gives a good picture into how generalized such large models are, and how well they are able to predict the next tokens for image/audio data at a high accuracy.
Do you mean the number of tokens in the LLM's tokenizer, or the dictionary size of the compression algorithm?
The vocab size of the pretrained models is not mentioned anywhere in the paper. Although, they did conduct an experiment where they measured compression performance while using tokenizers of different vocabulary sizes.
If you meant the dictionary size of the compression algorithm, then there was no dictionary because they only used arithmetic coding to do the compression which doesn't use dictionaries.
A detailed analysis of the DeepMind/Meta study: how large language models achieve unprecedented compression rates on text, image, and audio data - and the implications of these results
I don't know. I have found that the folks on Technology community appreciate many of my computer science posts. But a dedicated Comp Science community which is active, will be awesome.
Thank you! That's helpful. I spent quite some time trying to understand the difference between UTF-8 and Python's representation and arrived at the same understanding as you wrote. However, most of the external documents simply say that strings in Python are UTF-8 which made me conclude that perhaps I am missing something and it might be safer to write it as utf-8.
I will look more in the code as you suggested.
Hi @qwop, I am the author. Thank you for reading and the kind words. I would like to understand the error I made better so that I don't repeat in future, and if I can fix it. Could you please clarify?
I have the same problem. The number of things I want to read and write about is scaling faster than I can tackle them :)