Version 1.0!

I am extremely happy to announce htop 1.0!

Time flies, I can’t believe it’s been eight years of development already. It seems like yesterday that I’ve decided to stop writing PID numbers every time I wanted to kill a process and started this project. I am very happy to see this little project grow into a reality, see it being included in repositories for many distributions, reading nice reviews arond the web, receiving many contributions from coders from all over the world who helped making htop better and better over the years, and getting short “thank you!” emails that always make my day. Thanks to all distro packagers, reviewers, code contributors, users. The free software community is amazing; if it wasn’t for all of you, htop wouldn’t be what it is now. Version numbers are more symbolic than anything, but the stability of htop 0.9 in the past year and the cool new features introduced in this release compelled me to call this version 1.0. We all deserve this little “achievement”. :)

What’s new in version 1.0:

Performance improvements
Support for splitting CPU meters into two or four columns (thanks to Wim Heirman)
Switch from PLPA, which is now deprecated, to HWLOC.
Bring back support for native Linux sched_setaffinity, so we don’t have to use HWLOC if we don’t need to.
Support for typing in user names and column fields in selection panels.
Support for UTF-8 tree drawing (thanks to Bin Guo)
Option for counting CPUs from zero (thanks to Sean Noonan)
Meters update in every screen (no longer halting while on Setup, etc.)
Stricter checks for command-line options (thanks to Sebastian Pipping)
Incremental filtering (thanks to Seth Heeren for the idea and initial implementation)
Try harder to find the ncurses header (thanks to Moritz Barsnick)
Man page updates (thanks to Vincent Launchbury)
BUGFIX: Support larger numbers for process times. (thanks to Tristan Nakagawa for the report.)
BUGFIX: Segfault in BarMeterMode_draw() for small terminal widths (patch by Sebastian Pipping)

http://htop.sourceforge.net/index.php?page=downloads

Попаднах на доста добри новини – невероятният нов  init-демон systemd, за който доста се говори напоследък и който от известно време се използва като основен мениджър на процесите във Fedora, вече е интегриран и в любимия Debian! В момента най-новата версия на systemd е част от Debian Testing (Wheezy), което означава, че вече си е подсигурил мястото в следващия Debian Stable.

Systemd е новаторски init-демон, който, освен че е съвместим с класическия SysV и LSB-хедърите, предоставя възможности за “агресивно паралелизиране” на boot-процеса, както и изключително гъвкави и оптимални алгоритми и инструменти за управление на процесите в системата. Позволява следене на състоянието на процесите, както и групирането им с помощта на механизма cgroups, заложен в ядрото. Също така поддържа следене / автоматично монтиране и демонтиране на файлови системи, както и възможност за съхранение на снапшоти на системата и възстановяването и към предишно състояние.

С две думи systemd е изключително мощен и модернизиран инструмент, който се явява като естествен и многократно превъзхождащ по функционалност класическия init, като в същото време май вече започва да превъзхожда в много отношения (имайки предвид крехката му възраст) прехваления Upstart на Ubuntu.

Повече за самия systemd можете да прочетете от един от създателите му тук.

А тук можете да намерите повече информация за интеграцията на systemd в Debian Testing.

UPDATE – 06.12:

След инсталация, OpenSuSE чупи хард-диска! Вследствие на което, ако използвате AHCI контролер, вграден в дъното, той престава да вижда хард-диска ви и тотално забива, докато го засича. След това единственият начин да стартирате системата или дори да влезете в BIOS-а, е като изключите AHCI режима и преминете в Legacy mode. След зануляване на хард-диска, нещата се оправят и AHCI контролерът отново го засича нормално. Не знам какво прави OpenSuSE с хард-диска, но го омазва тотално. Пробвано е на две различни машини с два различни AHCI контролера и резултатът е един и същ.

Преди няколко дни излезе новата версия на OpenSuSE – 12.1 и вчера я инсталирах на виртуалка под Virtualbox, за да я тествам и да видя как се държи.

Освен всички добреизвестни новости, писани на хартия (linux kernel 3.1, Gnome desktop 3.2 и т.н.), огромно впечатление ми направи и нещо, за което не бях прочел никъде, а именно – системата е станала светкавично бърза! Не знам дали е заради новия systemd init-демон, който са започнали да използват от тази версия, или заради нещо друго… но фактите са неоспорими: от напълно зареден десктоп, през пълен reboot до логин екран – за 9 секунди! Измерено с хронометър. Това е невероятно постижение, особено за OpenSuSE, която досега не се славече особено много с бързина или лекота. Но новата версия на системата наистина работи адски пъргаво и ми се стори изключително лека.

Приятно впечатление ми направи също и новата десктоп-среда Gnome 3.2 с новия gnome-shell, който вече изглежда и се държи осезаемо по-стабилно отпреди. Явно са пооправили някой и друг досаден бъг.

Като цяло мога да кажа само хубави неща за новото SuSE. Много приятно ме изненада, особено по отношение на бързината и лекотата откъм системни ресурси (250 MB RAM заети при boot до десктоп) – определено това е нещото, което им куцаше преди и явно са го оправили. А ако се окаже, че новият systemd наистина е способен на такива чудеса, на каквито станах свидетел при boot-процеса, то тогава Upstart е вече история.

This is going to be long and rambling. If you’re going to read it, you may want to wait until you’re ill, and can’t get out of bed, and your head is filled with cotton, and you’re eating painkillers like they were candy. I don’t want you to feel pain while reading. Being unconscious and having a speech synthesizer read it to you at high speed is an even better option.

Linux is 20 years old this year. That’s a long time. Since I was there from the beginning I thought I’d share some memories of what’s happened.

In 1988 I graduated from high school, and got accepted into the University of Helsinki to study computer science. The studies started in September, and also in September I got invited to join Spektrum, the Swedish speaking club for those studying math, physics, chemistry, geography, or computer science.

Spektrum is a social club, which was good, since I was, and remain, shy and socially awkward, and the club provided me with a way to easily meet people when I’d moved into a new city. That’s also where I met the only other Swedish speaking new CS student of that year, a guy named Linus Torvalds.

That first year, we took some of the same classes, since all new students took those classes, and we met at Spektrum as well. A sort of friendship grew.

Computers were quite expensive back then, and the university provided access to classrooms full of PCs running MS-DOS, plus a few Macs, and some terminals connected to a big VAX/VMS system. I never liked MS-DOS that much, and they were often all in use. I couldn’t make heads or tails of the couple of Macs I tried, never having seen a GUI before. Thus I naturally graduated to the terminals, even though VAX/VMS was a horrible system to use, I thought.

After Christmas, things changed a bit. The CS department had a small Ultrix computer hidden away, mostly unused, and I happened to get access to that. Ultrix was DEC’s version of Unix. I had read about Unix, particulary in the K&R C book, and liked what I’d read. I had even written a few MS-DOS command line tools that worked like similar Unix tools. It was a joy to get access to a real Unix computer: pipes worked in real time, not via temporary files! Multiple processes at the same time! Filenames weren’t unnaturally constricted! It was quite liberating.

While playing around with the Ultrix box, which I think was called kreeta (Finnish for Crete, the island), one day I accidentally typoed the “rm” command. I had previously developed a habit of typoing “em”, which was the local version of MicroEMACS, as “rm”, so I tried very hard not to typo commands. However, that day, I typoed “rm something” as “rn something”, and discovered Usenet. Continue reading “Linux at 20, some personal memories” »

Най-накрая! Добри новини! :)

Днес на сайта на Debian официално обявиха новия релийз. Squeeze се появи на бял свят след 24-месечна непрекъсната разработка.

Новият релийз е съпътстван и от обновена визия на уебсайта.

Новината можете да прочетете тук.

Едва ли има някой Линукс потребител, за когото излизането на новия Debian Squeeze, версия 6.0, да не е голямо събитие. За мен това важи дори с още по-голяма сила, защото аз с огромно нетърпение чакам да обявят официално излизането на новия Debian, за да си го инсталирам вкъщи и да пенсионирам най-накрая старата си домашна инсталация на Ubuntu Hardy Heron, която вече е на достолепните две години и половина.

Както мнозина от вас може би знаят, Debian Squeeze в момента е във фаза RC1, след като мина през две алфи и две бети. В офиса я ползвам от алфа 2, но за вкъщи смятам да изчакам финалната версия на debian installer да излезе, преди да си инсталирам дистрото. Естествено, няма обявена официална дата, когато това ще се случи, и това е съвсем според традициите на проекта Debian и е типично за техния feature-based release cycle.

Squeeze беше замразен на 6 август миналата година и оттогава насам тече обичайната процедура по отстраняване на release-critical бъгове – или иначе казано, бъгове, които трябва да бъдат отстранени преди релийза. Макар да няма посочена дата, практиката сочи, че обикновено между датата на “замразяването” и датата на релийза на дадена версия на Дебиан минават около шест месеца. От тези сметки излиза, че можем да очакваме релийза на Debian 6 някъде февруари месец, ако всичко върви по план.

Междувременно нетърпеливите могат да си изтеглят LiveCD на последния snapshot на Debian Squeeze от сайта http://live.debian.net/, където ще намерите съответните линкове. Аз вече врътнах вкъщи LiveCD-то на RC1 и изтествах input-а на звуковата ми карта – нещо, което беше счупено и не работеше в Ubuntu 10.04 и заради което се отказах от това дистро – оказа се, че в Debian (както и очаквах) работи без проблем, може би заради по-новата версия на ALSA-та там.

И продължавам с голямо нетърпение да чакам новината за релийза на Debian Squeeze 6.0.

These days, the init.d script dependencies in Squeeze are quite complete, so complete that it is actually possible to run all the init.d scripts in parallell based on these dependencies. If you want to test your Squeeze system, make sure dependency based boot sequencing is enabled, and add this line to /etc/default/rcS:

CONCURRENCY=makefile

That is it. It will cause sysv-rc to use the startpar tool to run scripts in parallel using the dependency information stored in
/etc/init.d/.depend.boot, /etc/init.d/.depend.start and /etc/init.d/.depend.stop to order the scripts. Startpar is configured to try to start the kdm and gdm scripts as early as possible, and will start the facilities required by kdm or gdm as early as possible to make this happen.

Give it a try, and see if you like the result. If some services fail to start properly, it is most likely because they have incomplete
init.d script dependencies in their startup script (or some of their dependent scripts have incomplete dependencies). Report bugs and get the package maintainers to fix it. :)

Running scripts in parallel could be the default in Debian when we manage to get the init.d script dependencies complete and correct. I expect we will get there in Squeeze+1, if we get manage to test and fix the remaining issues.

If you report any problems with dependencies in init.d scripts to the BTS, please usertag the report to get it to show up at
<URL: http://bugs.debian.org/cgi-bin/pkgr…s-ng-devel@lists.alioth.debian.org >.

Happy hacking,
Petter Reinholdtsen

PAE really really sucks.

The biggest single reason to go 64-bit is exactly because of physical address space. Your virtual address space needs to bea multiple of the physical one: when you hit 1GB of RAM, 32-bit virtual memory is no longer acceptable. You literally do need more virtual memory than physical.

PAE turned that very simple fact on its head, and screwed things up royally. Whoever came up with the idea was totally incompetent, and had forgotten all the DOS HIGHMEM pains. There’s a damn good reason why we left the 286 behind, and started using 386′s, instead of having HIGHMEM crap with windows into a bigger physical space.

Repeat after me:

Virtual space needs to be bigger than physical space. Not “as big”. Not “smaller”. It needs to be bigger, by a factor of at least two, and that’s quite frankly pushing it, and you’re much better off having a factor of ten or more.

Anybody who doesn’t get that is a moron. End of discussion.

Reasons for why you need a bigger virtual space:

• you need to map that physical memory somehow, and no, tiny windows into the physical memory simply do not cut it! If you cannot have normal pointers to the physical space, it immediately means that you need to jump through serious hoops to get there.
• you additionally need to be able to remap things in alternate ways (ie user space) or make space for non-memory issues (virtual page tables, IO, you name it)

Ergo, a factor-of-two is a requirement. PAE was a total and utter disaster.

Yes, Linux supported it, and probably did so better than anybody else. But “better than anybody else” still wasn’t very good. Because you couldn’t use normal pointers to point to arbitrary physical memory, all the memory that couldn’t be accessed directly (ie anythign that didn’t fit in the virtual address map, which also had the user space memory in it) was basically limited to “special uses only”.

So you could allocate user pages in it, but you had huge problems with things like internal kernel data structures, which can be the bulk of your memory needs under some (not that unusual) loads. Directory caches, inodes, etc couldn’t use it, and in general it meant that under Linux, if you had more than 4GB of physical memory, you generally ran into problems (since only 25% of memory was available for normal kernel stuff – the rest had to be addressed through small holes in the tiny virtual address space).

I’m not at all surprised that Windows didn’t push PAE either. It was a total braindamage. I bet they supported it in the server offerings just because they had to, and I bet they did a much worse job of it than Linux did, and the reason you can do that with servers is that the loads are much easier, and you can expect maintainers to set magic config entries to tweak it to make it appear to work well for any particular load, when in reality it is fragile as hell and works only with duct-tape and prayers.

That kind of “duct-tape and prayers and lots of specialized knowledge about the load” is simply not possible in a desktop environment. Yeah, users have prayers, but they lack the duct-tape and the knowledge to work around the problems.

And dammit, in this age and date when almost everybody has a gigabyte of RAM in any new machine, anybody who still thinks that “not that many people need 64-bits” is simply not aware of what he’s speaking of.

Go back and play with HIGHMEM.SYS on a 286, and stop blathering crap. When you’ve spent the last ten years of your life working with HIGHMEM.SYS, then you can come back and tell me that we still don’t need 64 bits. Until that is the case, anybody who still doesn’t get why 64 bits is a requirement should just shut up rather than make a total fool of himself.

So repeat after me: PAE didn’t ever really fix anything. It was a mistake. It was just a total failure, and the result of hw engineers not understanding software.

So no, PAE does not mean that you can use more than 4GB of RAM. Even before PAE, the practical limit was around 1GB, and PAE didn’t move that post a fraction of an inch!

Linus

It was recently noted that ioctl() system calls are still executed with the Big Kernel Lock (BKL) held. A suggestion was made that drivers which can implement ioctl() without the BKL held should be specially flagged as a way of increasing parallelism. That suggestion looks like it will not get very far. But it did pique your editor’s interest in current use of the BKL. Besides, there hasn’t been a whole lot else going on this week.

The BKL is an artifact from when the Linux kernel first supported multiprocessor systems. Making the kernel safe for concurrent access from multiple CPUs has been a multi-year task; it is not a job that could have been done all at once at the beginning. So Linux 2.0 supported SMP systems by way of the BKL, which only allowed one processor to be running kernel code at any given time. The BKL is essentially a spinlock, but with a couple of interesting properties:

• The BKL can be taken recursively; the kernel remembers how many times a given thread has called lock_kernel() and does the right thing. Normal spinlocks are rather less forgiving.
• Code holding the BKL can sleep. The lock is released while the given thread sleeps, and reacquired upon awakening.

The BKL made SMP Linux possible, but it didn’t scale very well. Its overhead could be felt even with two processors, and it made running on anything larger problematic. So the kernel developers have been breaking the BKL into finer-grained locks ever since. Thus, for example, the block I/O subsystem went from the BKL to its own lock (io_request_lock) in 2.2, and from that to individual queue locks in 2.6. The kernel now has thousands of locks, and some people had assumed that the BKL would be gone by 2.6.

As it turns out, there are still over 500 lock_kernel() calls in the 2.6.6 kernel. For the curious, here are some of the places which still rely on this old, system-wide lock:

• The core kernel retains a few calls. The implementation of the reboot() system call is one of them; this is, of course, not one of the more performance-sensitive parts of the kernel. The boot-time early initialization process is also run with the BKL held. The sysctl() system call is run under the BKL; interestingly, while much of /proc is also implemented under the BKL, it appears that reads and writes to /proc/sys do not run with the BKL held.
• Many older filesystems (UFS, coda, HPFS, FAT, NCP, SMB, Minix, etc.) make heavy use of the BKL for serialization. The UnixWare “Boot File System” implementation has several calls; somehow, they seem unlikely to be fixed anytime soon. There are also lock_kernel() calls in NFS, UDF, isofs, the reiserfs journaling code, autofs, and some others. The ext2 filesystem uses the BKL to protect modifications to the superblock; ext3, instead, had all of its lock_kernel() calls purged during the 2.5 development process.
• The rpciod kernel thread spends its entire life with the BKL held.
• Core dumps are created with the BKL held.
• Block and character devices have their open() methods called under the BKL. Block release() methods are also called this way, but that is not true for char drivers. The default llseek() method runs under the BKL, but, if a driver or filesystem provides its own llseek() method, that method will not be called with the BKL held. The fasync() method is always called under the BKL. As noted at the beginning, ioctl() methods are called with the lock held; additionally, the ugly code which does 32-bit emulation on 64-bit systems needs the BKL.
• The file locking code still requires the BKL.
• Almost 10% of the lock_kernel() calls can be found in the (old, deprecated) OSS sound code. The ALSA code has no BKL calls, with one exception: the implementation of its /proc files.
• Most of the architectures retain some calls in the arch-specific code. The ptrace() system call is one common place for these calls. i386 also uses the BKL to protect llseek() calls on the CPUID and MSR pseudo-devices. uClinux performs execve() calls under the BKL.
• Almost all of the remaining BKL calls are to be found in device drivers. The TTY subsystem still has quite a few of them, as does USB. Many of these calls are protecting llseek() implementations. Quite a few of the rest are for the creation of special-purpose kernel threads: the daemonize() function needs to be called with the BKL held. Those calls can, presumably, go away as the driver code is (slowly) migrated over to the new kthread calls.

Given how poorly the BKL is viewed, it may be surprising that so many places in the kernel still use it. The simple fact is that, with regard to the BKL, all of the low-hanging fruit has long since been taken. For most of the remaining calls, removing the BKL is not worth the trouble and code churn. So, while removal of the remaining calls over the 2.7 development series looks entirely possible, it would not be surprising if that does not happen.

http://lwn.net/Articles/86859/