NAME
mirroring-ports —
how to build a mirror for ports
distfiles
DESCRIPTION
The OpenBSD Ports Collection offers some
powerful tools to mirror software sources.
dpb(1)
features a -F option which is explicitly designed
for mirroring distfiles.
If run with -F jobs,
dpb will
- Limit itself to fetching distfiles, and not build any packages.
- Disregard any architecture or broken annotation, and try to fetch every distfile.
- Fetch files to a temporary copy named
some_file.part using
ftp -Cto resume interrupted downloads. - Keep a global list of sha256 checksums as ${DISTDIR}/distinfo, and use that to refetch files when the ports tree records a changing checksum.
- Produces a log of old distfiles in ${DISTDIR}/history,
- Create sha256 links under ${DISTDIR}/by_cipher/sha256 as per link-checksum(1)'s former duties.
For partial distfiles collections, dpb can
also be run with
-DHISTORY_ONLY to scan the
full ports tree and update ${DISTDIR}/history
without fetching anything.
FILES
- ${DISTDIR}/distinfo
- a cache of known distfiles with their respective checksums.
- ${DISTDIR}/history
- List of files appearing in ${DISTDIR}/distinfo
that seem to no longer be required by the ports tree.
dpb(1) will
append to this file each time it is run on the whole tree (option
-a) and only if the ports tree scan finishes without error. Each line is of the formtimestamp SHA256 (file) = sha
denoting the first time a file/sha entry was no longer seen in the ports tree.
SEE ALSO
HISTORY
The new integrated -F option to
dpb(1) was
introduced in OpenBSD 5.1, replacing the original
infrastructure introduced in OpenBSD 2.7.
CAVEATS
Changing checksums is a recurring problem that is outside the
direct control of the OpenBSD Project. Some software
distributors change distribution files without warning, without changing the
file name proper. Once the problem has been identified, the port maintainer
should usually contact the software author to fix the problem or, if the
software author is unresponsive, the maintainer should use
DIST_SUBDIR to provide some state to guard against
shifting checksums.
However, a more robust approach is also needed, so that ports users can depend on distfiles mirrors to carry what they need irrespective of those synchronization issues. The ${DISTFILES}/by-cipher/sha256 directory provides more persistent access to the distfiles, indexed through the actual checksums that the files should match. Provided mirroring is run sufficiently often, two versions of the same distfile with respective checksums cksum1 and cksum2 will be available under the names ${DISTFILES}/sha256/c1/cksum1/distfile and ${DISTFILES}/sha256/c2/cksum2/distfile.
If REFETCH is set to true,
bsd.port.mk(5) will try to retrieve files under that naming scheme as
a last resort.