- Hmm
08/04/12: Things have changed tramatically over a year in the works, expect results when fully implemented / integraded in your O/S.
07/31/11: Put on the back burner for a short minute.
07/15/11: Cut through the mutter, i still suggest reading the below. 3 Jobs gone
05/22/11: If you do nothing, read the bottom 2 paragraphs (I promise you, you will want to read it all). Now we sit on 3 jobs gone
One at time of writing.A new tool for the masses... Well I suppose that is not the proper terminology. I am writing a tool to take away one of my jobs, so I can be more attentive to the forums and actually help people. I like to refer to it as programming automation at it's finest. It is being written with advanced users in mind. Ultimately, this program will change the *Nix world. One way or the other Eventually won't matter if you run any O/S minus "the other o/s", any nix base please read on.
Lets man up:
- Code: Select all
theemahn@SledgeHammer:$ man repomaster
Repomaster(1) Repomaster Repomaster(1)
NAME
repomaster - Repository Master
SYNOPSIS
repomaster -<-OPTION> [FOLDER|DEBNAME|DISTNAME|USERNAME@URL/DIST‐
NAME|ADD|REMOVE]
DESCRIPTION
repomaster - Software suited for extraction, packaging and maintaining
a repository.
FEATURES
Repomaster has many advanced features to automate the building of debs
to be Lintian error free. It has advanced scanning of files to set
proper permissions and a fix routine to scan for errors and auto cor‐
rect them. I have ran this tool on a folder with 13 GB of debs; prior
to running the tool a total of 136,435 errors and warnings. After run‐
ning the tool 1,323. Repomaster is not perfect yet, but is going in the
right direction. The numbers continue to diminish.
COMMAND SWITCH USAGE
Every 'flag' option has a 'no-flag' counterpart with exception of help.
Specifying no switch is the same as using the --help switch.
If an option or flag is marked as [XXX], it is optional. Options speci‐
fied with <XXX> are required. They will only work in combination with
the XXX option. Example: repomaster --service
Will not work because the format is [OPTION] <ADD|REMOVE> in this case
--service is the [OPTION] which is required as per Synopsis above. No
<required> switch was provided. IE repomaster --service ADD which is
actually case insensitive the | seperator signifies either, but not
both. On the other hand repomaster --extract is just fine without pro‐
viding a secondary switch. Please see OPTIONS below for switch require‐
ments and descriptions.
OPTIONS
Version
-v , --version
Display repomaster's version number.
Help
-h , --help
Shows help.
Build
-b , --build [FOLDER]
Builds specified deb. Specifying no folder name will build all
debs in current directory; prompting what will be built. All
error free debs are moved to a hidden sub-folder called .debs/
in the current folder. Logging individual errors to
[FOLDER].deb.errors and globally to all_errors.txt for further
review.
Extract
-e , --extract [DEB]
Extracts specified [DEB]. Specifying no deb filename will
extract all debs in current directory; prompting which debs will
be extracted.
Relaxation
-r , --relax [FOLDER]
Will relax the permissions on a given [FOLDER] allowing you to
edit or delete files within the folder. The proper permissions
will automatically be reset when you use the build switch.
Warning: Not specifying a folder will reset permissions on all
sub-folders. Please use care where you execute this tool. It
will display the folders to be effected and prompt you prior to
doing anything.
Clean
-c , --clean [FOLDER]
Will erase specified [FOLDER] ignoring permissions. Warning:
Specifying no folder name will erase all sub-folders in the cur‐
rent folder. Please use care. repomaster will prompt you of
folders to be effected. It will also erase all previous build
and error logs.
Fix
-f , --fix [FOLDER]
Will attempt to fix all errors in the specified [FOLDER] to be
Lintian / Debian error free and complaint to Debian standards.
Specifying no folder name will process all folders with a
[FOLDER].deb.errors that were generated by the build switch.
This will take a long time to be fully implemented.
Prepare
-p , --prepare <DISTNAME>
Prepares known good .debs/ for publishing. Builds the directory
structure in a folder called <DISTNAME>. Repomaster then GPG key
signs the debs as well as the release key for distribution. This
command switch also auto generates packages.gz for repository
management. Knowledge of how a repository works is highly recom‐
mended when using this command switch. Please man gpg for more
information on key signing.
Publish
-P , --publish <USERNAME:[PASSWORD]@URL/DISTNAME> Will publish known
good debs to remote address of your specification. Repomaster
will upload all files processed by the prepare switch associated
smartly. It will compare files on the remote server to local
host and only update appropriately, furthermore by architecture.
A future version will allow multiple distributions to be han‐
dled. The password is [optional] as ssh will prompt for it prior
to engaging rsync. Please see man ssh for more information on
how the command is passed. Knowledge of how a repository works
is highly recommended. when using this command switch.
Service
-s , --service <ADD|REMOVE>
Installs or removes based on second option to set repomaster as
a service. This feature is currently not implemented, however
when complete will install repomaster as a service. Once set as
a service repomaster will monitor any package being installed
and scan for errors and warnings optionally fixing them via end
users discretion. This powerful feature will allow the end user
to make the decision if or not errors and warnings enter their
system prior to the package even being installed. Similar to
what anti-virus software does.
Scan
-s , --scan
Version scans all debs to be processed. This should be the first
step you take when dealing with large amounts of debs. When pro‐
cessing a single deb this switch is useless. There is no sense
in building 2 seperate versions of debs. On the large scale this
is not a required , but recommended base of first action.
Verbose
-V , --verbose [FOLDER]
Initiates verbose mode. Invokes Lintian to verbosely report
errors and gives instructions on how to fix them. If I see this
project all the way through this switch may be come unnecessary,
time will tell.
PUBLISHING NOTES
Specifying the -p prepare or the -P publish switch I felt I should take
the time to document. Most people will not be interested in either
switch unless they intend to run a repository on-line or locally. Time
permitting, I will either write a on-line document and link here or
elaborate more in this section of the man page.
EXAMPLES OF REPOMASTER USAGE
TYPICAL USAGE IN ORDER
1. Scanning of deb(s) to ensure newest version (sometimes unnecessary)
2. Extraction of the deb(s)
3. Re-building of the deb(s) to obtain the error(s) and warning(s)
4. Scanning and fixing of errors(s) and warning(s)
5. Goto #3 until error(s) and warning(s) = 0
Repository maintainers will do the above followed by:
5. Prepare switch followed by repository name
6. Publish switch followed by remote address
EXAMPLES
Extract a deb:
repomaster -e ultamatix-1.9.3_all.deb
Re-build the deb:
repomaster -b ultamatix-1.9.3_all/
Fix errors:
repomaster -f ultamatix-1.9.3_all/
Extract all debs in your current folder into their own sub-folder:
repomaster -e
Prepare known good debs:
repomaster -p themes
Publish known good debs to remote server:
repomaster -P USERNAME:[PASSWORD]@Http://themelinux.com/themes/ themes
FILES
/bin/repomaster
Software for extraction, packaging and maintaining a repository.
/usr/share/man/man1/repomaster.1.gz
The Repository Master manual (this manual)
~/.gnome2/nautilus-scripts/RepoMaster/Extract Deb
GUI - Zenity based nautilus script front-end for extracting debs
to the repomaster.
~/.gnome2/nautilus-scripts/RepoMaster/Build Deb
GUI - Zenity based nautilus script front-end for building debs
to the repomaster.
~/.gnome2/nautilus-scripts/RepoMaster/repomaster
GUI - Zenity based nautilus script front-end command switchable
to repomaster.
SEE ALSO
gpg(1), rsync(1), ssh(1)
The full documentation for this tool is maintained as a Tex-info man‐
ual. If repomaster and the info program are properly installed at your
site, the command
info repomaster
should give you access to the complete manual including a menu struc‐
ture and an index.
COPYRIGHT
Copyright © 2011 Ultimate Edition Team.
This is free software; see the source for copying conditions. There is
NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR
PURPOSE, to the extent permitted by law.
BUGS
GNU repomaster home page: <Http://www.ultimateedition.info/>. E-mail
bug reports to: <[email protected]>. Be sure to include
the word repomaster somewhere in the Subject: field."
AUTHOR
TheeMahn <[email protected]>
A MESSAGE FROM THE AUTHOR
Who reads these man pages anyways? The entire purpose for me taking on
this project is so I could lose a few of my jobs. Repository management
and a few programming tasks in which it currently automates. I would
love nothing more then to say my goal is for repomaster to function to
the point where 0 errors and 0 warnings get through. Time however, is
not and has never been on my side. I have a slew of other projects
going on and I am but one Mahn. I hope you enjoy the software. Feel
free to send me a beer.
This man page was written and is maintained by TheeMahn.
Lintian error free you can count on it ;)
The Repository Master 2011-06-06 Repomaster(1)
Manual page repomaster(1) line 184/249 (END)
I am not impressed, it is extracting a bunch of deb files, no real magic here.
- Well now, that is nice. I want to see it automatically fix 130,000+ errors. Now that would be impressive.
Will the end user be interested in this tool probably not (minus a bulletproof O/S). Distro writers, programmers, repository maintainers or package maintainers for sure. I have been offered to have it copy-written for free and I am considering it. I have said many a time I would never charge the end user a penny, this is not being written for the end user. Does not mean you can not as the end user use it free. We have a ways to go before reality begins to sink in.
I will place test case studies here. The man and 130,000+ errors & warnings was ran across our very own repository. If I could ignore the package maintainers disregard for error & license agreement we would have a perfect O/S when I am done. Yes, I know I have made errors in some of those packages, but all of mine are fixed to perfection on themelinux.com's repository. 0 errors, 0 warnings. By running this very same tool. It is impossible for most to fathom the impact this tool will have when widespread and used by the individual who wrote their program / packaged it, O/S developer once set as a service (yes, it is coming).
What does this mean? I do not care what deb, rpm etc. you install it will scan for errors and fix them prior to installation, as per your prompt. I have left alot out in deliberation of what it currently does.
I will begin case study #1 our repository and screenshot by screenshot to show you the impact as well as repair. I do not want to up-play our repository, wait until you see case study #2.
CASE STUDY #1 (our repository):
PLEASE BE ADVISED, THIS COMMAND WILL DOWNLOAD ~13 Gigabytes of debs into a folder called: "www.downloadue.info"
EDIT8/14/2012: now 50GB and 403 forbidden, alot of bandwidth consumption involved.
- Code: Select all
wget -r -A deb www.downloadue.info/repo/
- Down it rolls. This will take a long minute, this is when I wish I had FIOS.
I stand corrected:
FINISHED --2011-01-27 00:17:49--
Downloaded: 663 files, 14G in 5h 11m 57s (790 KB/s)
theemahn@SledgeHammer:/media/Storage$
A high speed line is recommended, we have cloud based hosting at 15 Gigabit total, I have crap inet, even worse uploading. I will continue this later when the software is available for testing. Admin are the only ones that have it currently. I will show you what I refer to:
- Code: Select all
cd www.downloadue.info/repo/
lintian *.deb > ew.txt
Let me try and save you the time.
theemahn@SledgeHammer:/media/Storage/www.downloadue.info/repo$ lintian *.deb > ew.txt
internal error: command failed with error code 2
warning: could not unpack package to desired level
warning: skipping check of binary package ultimate-edition-theme-2.1
Can't call method "data" on an undefined value at /usr/share/lintian/checks/deb-format line 63.
internal error: cannot run deb_format check on package wormux
warning: skipping check of binary package wormux
theemahn@SledgeHammer:/media/Storage/www.downloadue.info/repo$
Results (I am saving you hours of downloading and Lintian, don't believe me, feel free to do so):
ew.txt.tar.gz- Nasty, please continue to pay attention.
- (1.49 MiB) Downloaded 646 times
open ew.txt and begin to sweat. Lintian is there to find errors, I am going to be there to fix them. If I let it roll as it is 130,000+ are gone. I would like to say my goal 0 errors / warnings. My main goal is to lose one of my jobs. Many can't understand the things I do, I have a multitude of jobs. I need to automate programming
sed may do that, the guy that wrote that I do not hold a candle to. I sed so I love Linux and pipes - I do not miss Windows, except for an occasional game. This will eventually go away. Games will come here. I will see to it. API is squat to me.I suppose I should press the issue, my bday is coming in a few days. I wrote my first program at 11, almost 30 years ago. I will be 40. I said almost
We will begin with as has been seen: repomaster -e
I am w/o providing software providing results. I am not about $, I would love to give the software away. People are against my mode of thinking. Software to me everything should be free. Many feel I should be paid, I do not. I do know what this will do, imagine this used as a service though apt, synaptic. Nothing, but perfection enters your system. I do not care if you run Debian, Ubuntu, the list goes on.
Extraction complete. Error(s) in extaction: please review ar.log.
ar: ultimate-edition-theme-2.1.4_all.deb is not a valid archive
theemahn@SledgeHammer:/media/Storage/www.downloadue.info/repo$
Thank you sir. Did we not see similar. Let's build. 130,000 errors / warnings are smoked right? We will not utilize the -f switch. Solely -b the build switch.
vertris_0.3.2-2~getdeb2_i386/
vuze_4.2.0.2-1~getdeb1_amd64/
warsow_0.42.dfsg1-1~getdeb1_i386/
warsow_0.50-1~getdeb2_amd64/
warsow_0.50-1~getdeb2_i386/
warsow-data_0.42-1~getdeb1_all/
warsow-dbg_0.50-1~getdeb2_amd64/
warsow-dbg_0.50-1~getdeb2_i386/
warsow-server_0.42.dfsg1-1~getdeb1_i386/
warsow-server_0.50-1~getdeb2_amd64/
warsow-server_0.50-1~getdeb2_i386/
warzone2100_2.2.1-1~getdeb1_amd64/
warzone2100-data_2.2.1-1~getdeb1_all/
warzone2100-dbg_2.2.1-1~getdeb1_i386/
warzone2100-music_2.2.1-1~getdeb1_all/
webilder_0.6.4-1~getdeb1_all/
webilder-gnome_0.6.4-1~getdeb1_all/
wesnoth_1.6.4-1~getdeb1_all/
wesnoth-all_1.6.4-1~getdeb1_all/
wesnoth-campaigns_1.6.1-1~getdeb1_all/
wesnoth-campaigns_1.6.4-1~getdeb1_all/
wesnoth-core_1.6.1-1~getdeb1_amd64/
wesnoth-core_1.6.4-1~getdeb1_amd64/
wesnoth-core_1.6.4-1~getdeb1_i386/
wesnoth-data_1.6.1-1~getdeb1_all/
wesnoth-data_1.6.4-1~getdeb1_all/
wesnoth-dbg_1.6.4-1~getdeb1_amd64/
wesnoth-editor_1.6.4-1~getdeb1_all/
wesnoth-music_1.6.1-1~getdeb1_all/
wesnoth-music_1.6.4-1~getdeb1_all/
wesnoth-server_1.6.4-1~getdeb1_amd64/
wesnoth-server_1.6.4-1~getdeb1_i386/
wesnoth-tools_1.6.4-1~getdeb1_all/
widelands_12-0~getdeb1_i386/
widelands-data_12-0~getdeb1_all/
wine-doors_0.1.3_all/
wine-doors_0.1.3rc1_all/
wireshark_1.2.0~repack1-1~getdeb1_amd64/
wireshark-common_1.2.0~repack1-1~getdeb1_amd64/
worldofpadman_1.1-0~getdeb1_amd64/
worldofpadman_1.1-0~getdeb1_i386/
worldofpadman-data_1.2-1~getdeb1_all/
worldofpadman-server_1.1-0~getdeb1_amd64/
wormux_0.8.3-1~getdeb1_i386/
wormux-data_0.8.3-1~getdeb1_all/
wxcam_1.0.4-1~getdeb1_amd64/
xcurs-0.5.1/
xmlcopyeditor_1.2.0.4-1~getdeb1_amd64/
xmoto_0.5.1-1~getdeb1_i386/
xmoto-data_0.5.1-1~getdeb1_all/
xqf_1.0.5+svn825-1~getdeb1_i386/
xye_0.8.0-1~getdeb1_i386/
yabause_0.9.6-0~getdeb1_i386/
yarssr_0.2.2-4_all/
yofrankie_0rev24-1~getdeb1_all/
zathras_0.0+r1-1~getdeb1_all/
zevv-tetris_1577-1~getdeb1_i386/
Build the above debs?
- Workin.
It will automatically fix anything that does not conform. I have not tossed the -f switch yet (this is what will upset people). Many will be mad, but I am here to fix issues. Eventually make it take one of my jobs right? I am sure I pissed many off by releasing what I am going to fix. Where I am going? A wake up call. Oh, that is right I have not released it yet
I have 50 + admin most are oblivious to the impact of this tool. I promise you when released will gather a whole bunch of attention. O/S writers: Please be mad at me. My job is one. I do not care, they will come see me as well. Most have seen anything I have written as just that, I am used to it. Let's see how this turns out. I want to point out I have never said I own. You may think differently. I do not have time to even address my own forum. I do see millions per day oh thats hits
I am sorry & please work with me. Oh, I am sorry, we are sites. A million here a million there I push the final ISO to sourceforge right and that is 32 servers. Do the math on that. Did they not interview me? Solid petabytes. I give them kudos every chance I can.- Let me toss in an Easteregg a little late for Easter ;) Is that Natty? Didn't they just release that today? Built & installed ;)
Final product, I don't think so. I want you to think about this, lets say I installed repomaster first built Ultimate Edition 3.0 (service wise) and purged it from the system, a super wow is what I see. I am going to try and do so. Much has changed since the easteregg. It will scan license agreements and I will deny any that are not GPL as an O/S builder. Imagine unleashed solid. Where you as the end user has the decision. 0 errors / warnings enter your system. This is the reason for the above.
When the option -p happens I am done with the tool. -p is publish and will automatically index the debs and rsync the results to our server. I should not say our server will be able to do the same no matter the server. It will be a Nasty tool
When I am done. Replace my job right? 3 jobs crushed when I get there. Building an O/S should be my only job right? No, I miss the forum. Watch me autonomously stop it all. The tool does not build O/S's. I have software that I am working on that does that as well. Please call me lazy now 06/01/11: Let me correct the above paragraph, I am working on the publish routine today, however I am going to write a service routine to prompt the end user if they want it used as a service. This will not fix errors currently in your system, however will prompt you on any new package installed whether or not to fix such said issues.
As I build Ultimate Edition 3.0, will prompt me so those said issues will not be there (will be the first package I install before building). Once done, I am saying Ultimate Edition will be the only bulletproof O/S on the planet. Feel free to view my output of the tools usage and begin to get a grasp of the impact it will have on posts I have made today below.
) and like I tell my kids, I may get old but I will never grow up! 
