AUTHOR: Zhou Peng(Chowroc) DATE: 2006-12-05 LICENSE: GNU Free Documentation License Version 1.2 SYNOPSIS: crablfs -- User Based Package Management System DESCRIPTION: Every package and its files belong to a system normal user, and a batch script to rebuild or migrate your personalized LFS/BLFS system by install all your packages all at once. ATTACHMENTS: http://cheeseshop.python.org/pypi?:action=display&name=crablfs https://sourceforge.net/project/showfiles.php?group_id=180695 PREREQUISITES: python-2.4 pexpect-2.1(optional) HINT: (1) Introduction: crablfs: User Based Package Management System which is written in python. Every package and its files belong to a system normal user, and a batch script to rebuild or migrate your personalized LFS/BLFS system by install all your packages automatically all at once. The goal of "crablfs" is to exert more control on packages that compiled from source, and reduce the costs of system migration and backup. The basic theory is: every package in the system has a corresponding user it belongs to, the installation actions(copy, mkdir, etc) will run by this user, thus all the files a package generated have the mode of this user. At the same time, all those users belongs to a 'install' group, and the key directories in the system such as /usr/local, /usr/local/bin, etc have g+s,o+t mode, and owned to the 'install' group, to make different packages create their own files and directories in those directories. There are many benefits to use this method. which can be looked up from the "LFS hints": More Control and Package Management using Package Users (v1.2) http://www.linuxfromscratch.org/hints/downloads/files/more_control_and_pkg_man.txt Based on this theory, the package manager "userpack" is a command line interface like rpm and paco, for finishing some necessary operations. It can install, remove, query a package, or list all packages of current system, or find the package owner of a specified file. the package name($pkgname-$version) will add to the list file (/usr/src/packages.list for default). Every package user has its own HOME dir, which is default to /usr/src/$pkgname. It will stores compressed source archives, all patches and supplementary files for installation, and installation profiles which contains the information of the package, such as user/group, source archive, patches, create time and those commands to build this package. By default, "userpack" use interactive mode, which will start a shell like command line, in which you type "cmd $command" to execute the corresponding command, and those commands will been recorded to "installation profile"(described before). This profile can be used to execute installation commands automatically in noninteractive mode. The script "crablfs" is a batch script write in python, it is another ALFS instance in other words, which make use of the automated ability of "userpack". It can use the list file(/usr/src/packages.list) mentioned before, to operate the compressed source archives and installation profiles you appointed automatically. And you can also specified a directory that has the same structure as $homepre (/usr/src) to build a ALFS system automatically. This means that you can migrate from an old system to new ones more easily -- and rebuild your personalized system more quickly. The essential differences among Linux distributions is the differences of the package management ways. As a distribution, LFS actually does not has a mature package manager itself, this of course roots to the flexibility of LFS, but for daily use, a mature package management system is necessary. It is recommended to use User Based Management in the BLFS document, but the LFS hints only give the basic principles which is very creative, but the shell scripts is not enough to form a self-consistency system. So I write one regardless of unveil my superficialness, a hope that my crude remarks may draw forth by abler people. Of course crablfs is not only used for LFS, it aims to those packages that build from source code, so it can used on other distributions, too. It's now available from: https://sourceforge.net/project/showfiles.php?group_id=180695 http://cheeseshop.python.org/pypi?:action=display&name=crablfs (2) install crablfs To make sure that crablfs also under the control of package management, to install by the following way to build the tools chain: # export PYTHONPATH=/opt/lib/python2.4/site-packages // sys.path will not be influenced by PYTHONPATH *** Now the pexpect package is optional *** # tar xfz pexpect-2.1.tar.gz # cd pexpect-2.1 # python setup.py install --prefix=/opt # cd .. # tar xfz crablfs-0.1.tar.gz # cd crablfs-0.1 # vi userpack.dirs *** define all key directories *** *** these dirs will belong to 'install' group.and have g+s,o+t mode *** /usr/local/ /usr/local/bin/ /usr/local/doc/ /usr/local/etc/ /usr/local/include/ /usr/local/info/ /usr/local/lib/ /usr/local/libexec/ /usr/local/man/ /usr/local/man/man1/ /usr/local/man/man2/ /usr/local/man/man3/ /usr/local/man/man4/ /usr/local/man/man5/ /usr/local/man/man6/ /usr/local/man/man7/ /usr/local/man/man8/ /usr/local/sbin/ /usr/local/share/ /usr/local/share/doc/ /usr/local/share/info/ /usr/local/share/locale/ /usr/local/share/man/ /usr/local/share/man/man1/ /usr/local/share/man/man2/ /usr/local/share/man/man3/ /usr/local/share/man/man4/ /usr/local/share/man/man5/ /usr/local/share/man/man6/ /usr/local/share/man/man7/ /usr/local/share/man/man8/ /usr/local/share/misc/ /usr/local/share/terminfo/ /usr/local/share/zoneinfo/ /usr/lib/python2.4/site-packages/ /usr/lib64/python2.4/site-packages/ // Adjust it for you own needs # python setup.py install --prefix=/opt # cd .. # /opt/bin/userpack init # /opt/bin/userpack install -f /tmp/crablfs-0.1.tar.gz crablfs-0.1 crablfs> cmd tar xfz crablfs-0.1.tar.gz crablfs> cmd cd crablfs-0.1 crablfs> cmd python setup.py install --install-scripts=/usr/local/bin crablfs> cmd cd .. crablfs> cmd rm -rf crablfs-0.1 crablfs> commit *** Now the pexpect package is optional *** # userpack install -f /tmp/pexpect-2.1.tar.gz pexpect-2.1 crablfs> cmd tar xfz pexpect-2.1.tar.gz crablfs> cmd cd pexpect-2.1 crablfs> cmd python setup.py install crablfs> cmd cd .. crablfs> cmd rm -rf pexpect-2.1 crablfs> commit # unset PYTHONPATH (3) package management: Manager is 'userpack'. a. install a package: # userpack install $pkgname-$version for example: # userpack install rxvt-2.7.10 you can classify: # userpack install meida.mplayer-1.0pre8 because the system support '.' for user name. The short way is: # upm i $pkgname-$version ****** NOTICE: * It is not true for LFS system,the shadow package that LFS use only support [-a-z0-9_] chars, so I wrote a shadow module to access /etc/passwd, /etc/group, /etc/shadow, /etc/gshadow directly. *. So far the classification function have not been tested strictly. ****** Under the conditions above, the default action will taken, that is: the compressed source archive and patches are considered to have been in the dir of HOME(/usr/src/$pkgname), other will you should specify it with argument: # userpack install -f /mnt/file/packages/rxvt-2.7.10.tar.gz rxvt-2.7.10 to make those archive and patches to be copied to HOME. This command will start an interactive command line, it is simple and only several command. In fact it is like a shell, you can type any 'shell' command to install the package, only need add 'cmd' before the command to make the 'userpack' records the commands. These commands will be stored into HOME/.config(installation profile). crablfs> cmd tar xfz rxvt-2.7.10.tar.gz crablfs> cmd cd rxvt-2.7.10 crablfs> cmd ./configure crablfs> cmd make crablfs> cmd make install crablfs> cmd cd .. crablfs> cmd rm -rf rxvt-2.7.10 crablfs> commit You should known that the current user is the package user, not root; and current working directory is the user's HOME, so there may be ownership problem(What is the purpose of User Based Package Management System). When you type commands, you can use 'list' to see what you have typed: crablfs> list 0, tar xfz rxvt-2.7.10.tar.gz 1, cd rxvt-2.7.10 2, ./configure 3, make 4, make install Use 'rollback' to clean up these, and 'del N' to delete specified command. If you want to execute a command but not want it to be recorded into the 'list', just 'do': crablfs> do ls You must 'commit/cmt' at last, thus all the commands will be recorded to the .config, the $pkgname-$version couple will been add to list file(/usr/src/packages.list for default), to indicate that the package has been installed. If there is a command error makes the installation failed, and the fault can't been clean up by yourself, just type 'quit' and the installation terminated, but the archive and patches will still be copied to HOME. If there is any other supplementary files: # userpack install -f /mnt/file/packages/MPlayer-1.0per8.tar.bz2 \ -p /mnt/file/packages/all-20060611.tar.bz2 \ -p /mnt/file/packages/Blue-1.6.tar.bz2 mplayer-1.0pre8 // all is mplayer's codecs.Blue is GUI Skin. If you have the installation profile before, you can use -a|--auto to install automatically: # userpack install -a rxvt-2.7.10 This means the HOME/.config exists, otherwise use -c|--profile: # userpack install -ac /mnt/packages/confiles/rxvt-2.7.10 rxvt-2.7.10 If you have burned /usr/src to DCD-R's sources dir, you can also execute the automatically installation below: # userpack install -as /mnt/dvdrom/sources rxvt-2.7.10 Moreover, the sources could also on the net such as ftp or ssh: ssh://localhost/sources/mlterm-2.9.3.tar.gz scp://localhost/sources/mlterm-2.9.3.tar.gz the 2 above is same, ftp://localhost/pub/sources/mlterm-2.9.3.tar.gz for example: # userpack install \ -f ftp://192.168.0.1/pub/sources/mlterm-2.9.3.tar.gz mlterm-2.9.3 ****** If the package name(mlterm-2.9.3) is not given, it can be parsed from the archive name(mlterm-2.9.3.tar.gz) automatically by 'userpack'. ****** If a archive has problem and makes installation failed, the archive still has been copied to HOME, and the next time 'userpack' will check whether the archive exists in HOME, if true, copy will not occur. To change this, use -C|--copy-force: # userpack install -Caf libesmtp-1.0.3r1.tar.bz2 About the user/group ID, 'userpack' can define its own range, default from 1000 to 20000, U can define in /usr/lib/python2.4/site-packages/crablfs/shadow.py: UID_MIN 1000 UID_MAX 20000 GID_MIN 1000 GID_MAX 20000 ADM_MIN 950 ADM_MAX 1000 This mean the normal user ids range is (1000, 19999), admin group ids is (950, 999). So 'install' group will be 950 by default. Besides the 'install', other admin groups will be used as 'application groups'. For example, 'glib2', 'atk', 'gtk', 'gtk2' all belongs to 'GTK' group for some limited directories sharing, such as /usr/share/gtk-doc/ /usr/share/gtk-doc/html thus no modification of userpack.dirs is necessary. Correspondingly, add a '-G' option to support multiple groups, for example, 'xfce4' belogs to 'xfce4', 'install' and 'GTK' group. (default is 'xfce4' and 'install'). Now I can do this to support 'applications group' and 'multiple groups': # upm i -G GTK glib2-2.6.4 crablfs> ... ...... crablfs> chgrp GTK /usr/share/gtk-doc/{,html} crablfs> chmod g+w,o+t /usr/share/gtk-doc/{,html} ...... # upm i -G GTK atk-1.9.1 ...... These all can be read from the installation profiles and be executed automatically. b. remove: # userpack remove rxvt-2.7.10 or # upm r rxvt-2.7.10 c. list all packages under the control: # userpack packs or # upm a rxvt-2.7.10 mplayer-1.0pre8 crablfs-0.1 pexpect-2.1 mlterm-2.9.3 d. query a file's package owner: # userpack owner /usr/local/bin/mlterm or # upm o /usr/local/bin/mlterm mlterm-2.9.3 e. list all files a package contains: # userpack files pexpect-2.1 or # upm l pexpect-2.1 /usr/lib/python2.4/site-packages/fdpexpect.py /usr/lib/python2.4/site-packages/pxssh.py /usr/lib/python2.4/site-packages/pexpect.py /usr/lib/python2.4/site-packages/pxssh.pyc /usr/lib/python2.4/site-packages/pexpect.pyc /usr/lib/python2.4/site-packages/fdpexpect.pyc This will read $HOME/.files to print all files the package have. This file is generated by the Package Manager 'userpack' as soon as the installation is finished, but you can force 'userpack' to regenerate it by run: # upm L pexpect-2.1 This command will also print all the files of the package. (4) About the pkgname & version rule: This is the argument to 'userpack', the form is '$pkgname-$version', the python re pattern is: '^(?P(\w+-{0,1})+)(-(?P(\d[\w]*[-\.])*\d[\w]*))' That is '$pkgname' part is 'part1[-part2][-part3][...]', every part could contains '[-a-zA-Z0-9_]', no rules for start char, but it is recommended to use alphabet; when meet '-[0-9][a-zA-Z0-9_]\.' pattern('-' follows a string start with digit, and follows a '.'), this will be considered as start part of $version, $version can also contains other '-', and when meet a string not start with digit, $version ends. chars follows are ignored. (5) ALFS It is mentioned before that all commands will be recorded to HOME/.config, and 'userpack' can also use this commands to run automatically, so there could be a batch script to use packages.list to install those packages all at once. Based on this, we can get an instance of ALFS. The target of the instance is extended to tranditional nALFS and jhalfs, for more personalized environment, for example: The first time, I build LFS/BLFS step by step, and the packages installed, related with their own's information & operations are recorded to the profiles, the next time, I can use those profiles to build the software envrionment I need with a 'turn key' way, exactly as those before. This means I can migrate many times, and others can also use my profiles to build their own personalized system. The end target also contains a platform independent operation. This batch script is 'crablfs'. ****** So far, it is only tested under the condition that the LFS base system has been built, to use 'crablfs' to build a bunch of BLFS packages, so it is the future to make the LFS base system under the control of package management system. ****** At first, adjust the tool chain: # cd /blfs-sources # tar xfj python-2.4.1.tar.bz2 # cd python-2.4.1 # patch -Np1 -i ../python-2.4.1-gdbm-1.patch # ./configure --prefix=/opt --enable-shared # make && make install # cd .. # rm -rf python-2.4.1 // U'd better delete it, otherwise may be problems # export PYTHONPATH=/opt/lib/python2.4/site-packages *** Now pexpect is optional *** # tar xfz pexpect-2.1.tar.gz # cd pexpect-2.1 # /opt/bin/python setup.py install --prefix=/opt # cd .. # rm -rf pexpect-2.1 # tar xfz crablfs-0.1.tar.gz # cd crablfs-0.1 # cp userpack.dirs.blfs userpack.dirs // Adjust for you own need and system // Whenever U meet pemission problem, just adjust /etc/userpack.dirs // and run 'userpack init' # /opt/bin/python setup.py install --prefix=/opt # /opt/bin/userpack init # /opt/bin/userpack install -f python-2.4.1.tar.bz2 -p # python-2.4.1-gdbm-1.patch python-2.4.1 crablfs> cmd tar xfj python-2.4.1.tar.bz2 crablfs> cmd cd python-2.4.1 crablfs> cmd ./configure --prefix=/usr --enable-shared crablfs> cmd make crablfs> cmd make install crablfs> cmd cd .. crablfs> cmd rm -rf python-2.4.1 crablfs> commit # /opt/bin/userpack install -f crablfs-0.1.tar.gz crablfs-0.1 crablfs> cmd tar xfz crablfs-0.1.tar.gz crablfs> cmd cd crablfs-0.1 crablfs> cmd python setup.py install --install-scripts=/usr/local/bin crablfs> cmd cd .. crablfs> cmd rm -rf crablfs-0.1 crablfs> commit *** Now the pexpect is optional *** # userpack install -f pexpect-2.1.tar.gz pexpect-2.1 crablfs> cmd tar xfz pexpect-2.1.tar.gz crablfs> cmd cd pexpect-2.1 crablfs> cmd python setup.py install crablfs> cmd cd .. crablfs> cmd rm -rf pexpect-2.1 crablfs> commit # unset PYTHONPATH # userpack packs python-2.4.1 crablfs-0.1 pexpect-2.1 ****** Now there is a script 'scripts/setup-blfs' to do these all! ****** And use the same way, I installed some packages: # userpack packs # cat /usr/src/packages.list python-2.4.1 crablfs-0.1.1 pexpect-2.1 net-tools-1.60 openssl-0.9.7g cracklib-2.8.3 Linux-PAM-0.80 iptables-1.3.3 gnupg-1.4.1 pcre-6.1 libxml-1.8.17 libxml2-2.6.20 libxslt-1.1.14 gdbm-1.8.3 pkg-config-0.19 glib-1.2.10 glib2-2.6.4 expat-1.95.8 libesmtp-1.0.3r1 lzo-2.01 libusb-0.1.10a libjpeg-6b libpng-1.2.8 which-2.16 unzip-5.52 zip-2.32 pciutils-2.1.11 cpio-2.6 libpcap-0.9.3 ppp-2.4.3 rp-pppoe-3.8 libsoup-2.2.3 subversion-1.1.4 wget-1.9.1 bind-9.3.1 gc-6.8 w3m-0.5.1 gpm-1.20.1 dhcpcd-1.3.22 freetype-2.1.10 fontconfig-2.3.2 Xorg-6.9.0 915resolution-0.5.2 gtk-1.2.10 pango-1.8.1 atk-1.9.1 gtk2-2.6.7 startup-notification-0.8 hicolor-icon-theme-0.8 xfce-4.2.2 mlterm-2.9.3 The X Window is OK and xfce is usable, so I try to *migrate*: # mv /usr/src/packages.list /blfs-sources/ # copy-profiles /usr/src // This will generate a profiles dir in current dir, // which contains all the profiles from $homepre(/usr/src) # mv profiles /blfs-sources # crablfs -t alfs \ -C /blfs-sources/profiles/ \ -F /blfs-sources/ \ /blfs-sources/packages.list If you have installed such a BLFS system and burned /usr/src all to the DVD-R, you can do the following for another system to install all package once a time: # crablfs -t crablfs[/default] \ -s /mnt/dvdrom/sources/ \ /mnt/dvdrom/sources/packages.list You may want to adjust 'packages.list' for your own needs, if you want , you can also edit the profiles. crablfs will record current successfully installed package name to /var/log/crablfs/.mark. Because of the packages dependencies problems, if one package failed, the 'crablfs' process terminated. After you clean up the problem, you can continue from the breakpoint. So if .mark point to the last item of packages.list, 'crablfs' will do nothing. Adjust it for you need. Moreover, 'crablfs' will check packages.list, illegal package name will terminate 'crablfs' and do nothing; if the name in .mark not in packages.list, you will get a "x not in list" error report, you can adjust corresponding, or just delete the .mark file. So far, there are few permission problems, and all can avoid by changing the installtion arguments. The wrappers for chmod, chgrp, chown, install may be needed, but so far I have not see the neccessary. To dicide this for further using and testing. (6) The next plan * Add upgrade part * Add gettext internationlized support * Whole infomations show of a package * PyUnit * Add code for cmdline to read .config history commands * Make cmdline can execute 'root' commands, as 'sudo', by IPC? * A command editor? * File lock of packages.list * Change .config to the form of '$pkgname-$version' * Enganced platform independent support: Add var and env var to cmdline for different $ARCH & $LOCALE * Support multi installation sources * Enganced classification mechanism * Intergrated checking * Enganced configuration getting & setting: Plain Tree & XML * Consider package dependencies problems * Better documents: internal, design doc and manual * Make 'userpack init' can adjust those dir items deleted from 'userpack.dirs' ****** *** LIMITATIONS *** When want to install 'glib2-2.6.4', but type in error with 'glib-2.6.4': # upm i -a -c profiles/glib2 glib-2.6.4 This will mistakenly remove all files of glib-1.2.10!!! Because when a package installation failed, the Manager will remove all the files that generated by 'install', 'cp', 'mv' commands during the process. So far, this limitation roots in the limitation of package version control, this also makes 'upgrade' not easy. ****** ACKNOWLEDGEMENTS: Matthias S. Benkmann built the original principle CHANGELOG: 2006-12-05: For crablfs-0.1.1beta. 2006-10-25: crablfs-0.1 has been distributed at 2006-10-20, so I write this hint.