Updating selected debian/ubuntu repositories

I’ve been dealing with the pains of a slow connection for sometime now and one of the many annoyances is trying to update a single repository (in my case a launchpad ppa).  There are over 50 repositories that I subscribe to, which isn’t a problem so much, since most of them are updated infrequently and have only a few packages in them.  But since the official repos are large (like over 4Mb each) and updated frequently, I almost alway need to update them.  Even though downloading the official and unofficial repos happen in parallel, getting the official ones consume a significant percent of my bandwidth.  Also, I want to keep the previous repos not being updated (like the official ones).

After a little googling, I found others trying to solve this problem.  But the answers weren’t really satisfactory.  The simplest solution was to use an infrequently updated mirror.  While definitely something that I wanted to do, I still don’t want to be forced to update when this mirror gets updated, even if its once every two or three weeks.

One suggested ubuntu repository management tool to deselect undesired repos and then running update.  But this is a pain with over 50 repos, where most I don’t want to update.  Also by default unselected repos are deleted.  Since most 3rd party repos have dependencies on packages in the official repos, we don’t want to have the official repos removed.  The latter behavior is controlled by an apt configure option, but I like that behavior in general (when I have a fast connection).

The best solution I saw overrode config options of apt to have it use a file with the desired repo lines.  The original answer didn’t include using the options ‘-o APT::Get::List-Cleanup="0"‘ so that existing repos wouldn’t get removed.  Still this solution didn’t provide for multiple files of repo lines or specifying repos on the command line.

Still I wanted a general tool that could be given a series of files with repo lines and even have repo lines specified on the command line.  Well that and I needed an excuse to see if I could do what I wanted to with python-apt, the python binding to libapt-pkg.  So here’s what I came up with, and remember that python-apt must be installed for this to run properly:

	#!/usr/bin/env python
	# Copyright (C) 2012 Glenn Washburn
	# Permission is hereby granted, free of charge, to any person obtaining a copy
	# of this software and associated documentation files (the "Software"), to deal
	# in the Software without restriction, including without limitation the rights
	# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
	# copies of the Software, and to permit persons to whom the Software is
	# furnished to do so, subject to the following conditions:
	# The above copyright notice and this permission notice shall be included in
	# all copies or substantial portions of the Software.
	# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
	# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
	# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
	# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
	# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
	# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
	# SOFTWARE.
	# TODO:
	# * Write a more informative progress class that says which repo is being
	# updated.
	# * Add a gtk gui for easy selection/deselection
	# libapt-pkg documentation at:
	# file:///usr/share/doc/libapt-pkg-doc/html/index.html
	# A simpler form of what we do here:
	# apt-get update -o Dir::Etc::sourcelist="sources.list.d/other.list"
	# -o Dir::Etc::sourceparts="-" -o APT::Get::List-Cleanup="0"
	import sys
	import os
	import getopt
	import apt
	import apt_pkg
	class PrintableSourceList(apt_pkg.SourceList):
	def __str__(self):
	return '\n'.join(self.all_urls())
	def all_urls(self):
	for item in self.list:
	for index in item.index_files:
	yield index.describe
	class CustomAcquireProgress:
	pass
	def acquire_progress_factory(ui='text'):
	mod = getattr(__import__("apt.progress", fromlist=[ui]), ui)
	if ui == 'gtk2':
	base = mod.GAcquireProgress
	else:
	base = mod.AcquireProgress
	mixin = CustomAcquireProgress
	class AcquireProgress(mixin, base):
	pass
	return AcquireProgress
	def get_repositories(repos):
	# Create a temporary sources.list file to load the SourceList object from.
	# We can currently only create a SourceList objects by reading the
	# sources.list or its partial directory defined in the config. So we
	# set the setting to a tmp file in which we stuff the desired repo entries.
	path = '/tmp/reloadsources.list.%d'%os.getpid()
	fsrclist = open(path, 'w')
	# create fake empty sources.list.d
	dirpath = '/tmp/reloadsources.list.%d.d'%os.getpid()
	if not os.path.exists(dirpath):
	os.mkdir(dirpath)
	for repo in repos:
	# If repo is a file path, then assume is a sources.list fragment
	if os.path.isfile(repo):
	print >> fsrclist, open(repo).read()
	else:
	# Assume is a line in sources.list
	print >> fsrclist, repo
	fsrclist.close()
	# Tell apt that it should use the new sources.list.
	apt_pkg.config.set("Dir::Etc::sourcelist", path)
	apt_pkg.config.set("Dir::Etc::sourceparts", dirpath)
	#~ print apt_pkg.config.get("Dir::Etc::sourcelist")
	#~ print open(apt_pkg.config.get("Dir::Etc::sourcelist")).read()
	srclist = PrintableSourceList()
	srclist.read_main_list()
	os.rmdir(dirpath)
	os.unlink(path)
	return srclist
	def main(argv):
	verbose = False
	ui = 'text'
	opts, args = getopt.gnu_getopt(argv, "vu:")
	for opt in opts:
	if opt[0] == '-v':
	verbose = True
	if opt[0] == '-u':
	ui = opt[1]
	# ensure that other lists are not cleaned up (removed) because we only
	# want to update the ones we specify.
	apt_pkg.config.set('APT::Get::List-Cleanup', '0')
	# Get desired repos to update as a sourcelist
	srclist = get_repositories(args)
	if verbose:
	print srclist
	# Lock pkg dir so no one else is updating or trying to read
	with apt_pkg.SystemLock():
	cc = apt_pkg.Cache(None)
	AcquireProgress = acquire_progress_factory(ui)
	cc.update(AcquireProgress(), srclist)
	if __name__ == "__main__":
	main(sys.argv[1:])

view raw

reload_repository.py

hosted with ❤ by GitHub

Its slightly annoying that there doesn’t seem to be a way to add repos to a SourceList object directly (libapt-pkg seems not to support it either. So I have to write a temp file with all the repo lines in it.