From: eLinux.org
Inspiration for this project originated in experiments conducted by Timothy Miller in 20031(#cite_note-1). His experiments lead to the assumption that printk strings in the Linux Kernel can be compressed by 50% (~4% of the total size). No code or patches have been posted, yet the description of his ideas and the published codebook allow a discussion of his approaches, and of general challenges with this topic.
.i
-filesFurther notes:
allyesconfig
was used for the testsProblem: Find all printk-strings
<new_subsys>_dev_err, ...
Here is an example how printk was modified to put the string into a seperate section.
Define a wrapper:
+#define __printk(fmt, args...) \
+do { \
+ if (__builtin_constant_p(fmt)) { \
+ static const __attribute__((section("__printk"))) \
+ char __f[] = fmt; \
+ printk(__f, ##args); \
+ } else \
+ printk(fmt, ##args); \
+} while (0)
And apply it (one example here):
#define pr_emerg(fmt, ...) \
- printk(KERN_EMERG pr_fmt(fmt), ##__VA_ARGS__)
+ __printk(KERN_EMERG pr_fmt(fmt), ##__VA_ARGS__)
#define pr_alert(fmt, ...) \
- printk(KERN_ALERT pr_fmt(fmt), ##__VA_ARGS__)
+ __printk(KERN_ALERT pr_fmt(fmt), ##__VA_ARGS__)
Similar approaches have been implemented for dev_*
and BUG/WARN
. However, due to various side effects, a full kernel build could not be achieved in the timeframe for this project.
However, since the own section approach touches many files close to the core in nasty ways and will most probably cause side-effects, it is extremely unlikely to be accepted upstream. Same goes for the original approach scanning the intermediate files.
However:
allyesconfig
is unrealistic for tiny systemsBrainstorming:
Tokenization and BPE need unused characters for their symbols which might collide with UTF8 encoding. While UTF8 also has 'illegal encodings' which could be hijacked as compression symbols4(#cite_note-4), this is approach is hackish and will also decrease the compression factor by 50% (compression symbols are then 16 bit instead of 8).
Reusing unused ASCII characters as compression symbols is technically sensible here. However, giving up UTF8 cleanliness will probably be not well received upstream.
In the original implementation, the source code was piped through a filter on the second build of the kernel. Not being able to see what was actually compiled is expected to raise eyebrows when upstreaming.
Compressing the special printk sections turned out to be rather easy with recent binutils. However, updating all references to the strings is expected to be complex. In this timeframe, it was not possible to research the topic. However, heavily modifying object files as a post-processing step, maybe with modified binutils, is expected to have serious problems upstream.
All solutions to the problems investigated here turned to be very hackish already at the drafting phase. The likeliness of going upstream is close to zero. They are also no good candidates for keeping them out-of-tree since they tend to be very fragile when it comes to kernel updates.
It is worth noting that printk strings are only a subset of all strings in the kernel. For example, devicetree uses a lot of strings and keeps adding more. They might be easier to tackle since they are largely accessed via of_*
functions, but this will also add more complexity. To be worth this effort and receive more gain, the problem should be reevaluated at a higher level, considering all strings, maybe even all .rodata.
From: Managing Gigabytes, Witten/Moffat/Bell, 1st edition, p. 385:
We find ourselves in the midst of a practically important and intelectually fascinating convergence between the desire for more and better compression and the need to learn about what 'structure' there is in data.
So, trying to understand the structure and improve from there:
Proposal:
Already applied examples:
devm_ioremap_resource()
(unifies error handling for devm_ioremap
et al.)Further possibilities:
devm_get_optional
are also good candidatesSimple tests:
devm_clk_get
saved 20K instantlydev_*
and friends if possibleHow pr_fmt
gets applied:
#define pr_alert(fmt, ...) \
printk(KERN_ALERT pr_fmt(fmt), ##__VA_ARGS__)
How to simplify it:
-#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+#define pr_fmt(fmt) "%s" fmt, KBUILD_MODNAME ": "
That saved around 15% (or 250 byte) for sn9c20x. Applied to all 900 instances in the kernel, it saved about 30K (or 0.4%). It works better in some places than in others.
Redefine subsystem printouts:
/* UBI error messages */
-#define ubi_err(fmt, ...) pr_err("UBI error: %s: " fmt "\n", \
- __func__, ##__VA_ARGS__)
+#define ubi_err(fmt, ...) printk("%s%s: " fmt "\n", \
+ KERN_ERR "UBI error: ", __func__, ##__VA_ARGS__)
That saved around 15% (or 2.5K). UBIFS, JFFS2, SCSI layer seem also to be promising candidates.
Some code duplicates strings too easy5(#cite_note-5):
switch (sd->sensor) {
case SENSOR_OV9650:
ov9650_init_sensor(gspca_dev);
if (gspca_dev->usb_err < 0)
break;
pr_info("OV9650 sensor detected\n");
break;
case SENSOR_OV9655:
ov9655_init_sensor(gspca_dev);
if (gspca_dev->usb_err < 0)
break;
pr_info("OV9655 sensor detected\n");
break;
case SENSOR_SOI968:
soi968_init_sensor(gspca_dev);
if (gspca_dev->usb_err < 0)
break;
pr_info("SOI968 sensor detected\n");
break;
/* ... 7 more ... */
And in the init functions:
if (gspca_dev->usb_err < 0)
pr_err("OV9650 sensor initialization failed\n");
...
if (gspca_dev->usb_err < 0)
pr_err("OV9655 sensor initialization failed\n");
...
if (gspca_dev->usb_err < 0)
pr_err("SOI968 sensor initialization failed\n");
...
A little love helps a lot here <3
For example, here:
Be aware when adding strings: