Mirror Infrastructure

Эта статья содержит фрагменты на иностранном языке. Вы можете помочь переведя её до конца. (cм. руководство по переводу)

Этот документ описывает различные пути как продублировать содержание ftp.opensuse.org и как можно получить статус официального зеркала.

Что нужно, чтобы получить статус официального зеркала?

• Потребуется свободное место на диске в диапазоне, по крайней мере, 40-60 Гб. В зависимости от того, что зеркалируется.

• Также поддержка зеркала потребует расхода интернет-трафика. Достаточно сложно спрогнозировтаь его объём, но, например, в Германии это значение легко достигает 1 ТБ в месяц. Гораздо лучше, если зеркало в состоянии отдать 2 ТБ в месяц. Минимальные требования к каналу - 10 Мбит/с, а лучше - 100 Мбит/с. В некоторых регионах ситуация может существенно отличаться от указанной, соответственно, предъявляемые требования будут изменяться в зависимости от расположения. Вообще, чем больше контента будет зазеркалировано, тем больше вырастет интернет-трафик, с другой стороны, мы достаточно хорошо можем контролировать число редиректов. Наличие ISO-образов является наиболее существенной причиной снижения трафика.

• Аппаратная платформа зеркала или установленная на нём операционная система не имеют никакого значения.

Текущий размер модулей для rsync приведен тут: Mirror_Infrastructure#rsync modules Следует также отметить, что возможно зеркалировать только части модуля.

Rsync-сервер

Публичный доступ: rsync.opensuse.org

Данный rsync-сервер доступен каждому. Предоставляется публичный доступ по протоколу rsync к контенту на download.opensuse.org. Обычно доступ ограничивается 50 конкурирующими соединениями, таким образом, Вы сможете получить к нему доступ не всегда. Часть зеркал, приведенная тут также может предоставлять доступ к сервису rsync.

Доступ к зарегистрированным зеркалам: stage.opensuse.org

[RU] Зарегистрированные зеркала получают доступ к stage.opensuse.org. Этот сервер предоставляет доступ к обновленному контенту download.opensuse.org до того, как состоится официальный релиз, а также располагает большей скоростью передачи нежели публичные сервера. Вы можете отправить заявку на регистрацию на mailto:ftpadmin@suse.de в том случае, если ширина канала, к которому подключено Ваше зеркало, составляет как минимум 100 МБит/с и соблюдены условия, приведенные в параграфе ниже.

[ENG] Registered mirrors get access to stage.opensuse.org. This server provides the updated content of download.opensuse.org before the official release and has a higher transfer rate than the public servers. You may want to register for access at mailto:ftpadmin@suse.de, if your mirror has at least a 100MBit connection, and if the conditions outlined in the following paragraph are met.

Условия получения доступа к stage.opensuse.org

A few words about "staged content" up front. Staged content is content that is not meant to be public yet -- but which we still would like to spread to mirrors already, so that at the time of the public release it is already mirrored, and thereby accessible for many people. So how can that be achieved? We set the permissions of the directory to be protected to 'rwxr-x---' (0750). The directory is then served as part of the tree which is hosted on the stage rsync server. When mirrors sync from it, they will replicate those permissions. And when the to release has come, the directory permissions are changed to rwxr-xr-x (0755), and when the mirrors sync the next time, they catch up with it and the directory becomes accessible on their HTTP/FTP servers as well. This process of release by permission change is sometimes called "bit flip release".

There are some caveats with that, which you (as mirror admin) need to observe:

• run rsync with -p (--perms), so that the permissions are reproduced on the target machine.

• if you run a public rsync server: make sure that your rsync daemon runs under a different user id than the script which pulls the content. Otherwise you might be publicly serving the staged content. You can achieve this, for instance, by setting uid = nobody and gid = nogroup in the respective rsync module.

• run your mirror scripts under a user id different from the one which your HTTP/FTP server runs as. An identical user id would make all files readable for the the HTTP/FTP server. The same effect happens if you run the server as root.

• never run your web server (FTP server / rsync server) as root. A somehow recurrent misconfiguration is, if lighttpd is used, that it is run as root, because the configuration which causes it to run as a different user/group has been forgotten.

You should be subscribed to the mirror mailing list (see bottom of this page), so we can keep you up to date with regard to ongoing release activities. We will inform you of the release schedule, and exact timing of public release -- and you can actively support us in fact.

Registering Your Mirror

In order to redirect clients to your mirror, we need the following:

• email address for contact
• HTTP URL on your mirror (e.g. http://yourmirror.example.com/pub/opensuse/)
• is your web server large-file capable? (to handle images larger than 2 GB in size)
• read-only rsync access for our scanner -- for scanning which we perform to keep our download redirector database up-to-date. It is done from 195.135.220.0/22.
• FTP URL, if you run an FTP server. Can serve as fallback protocol for scanning, if rsync is not available. Otherwise, FTP is not used by openSUSE.
• a name and URL of the operator or sponsor of the mirror, for display in the mirror list.

If you provide this data in writing to mailto:ftpadmin@suse.de, we will add your mirror to our mirror database. The mirror database is used by our download server to actively redirect clients to your server. We attempt to distribute requests on a geographical basis per client IP address. The amount of redirects issued also depends on a score which we will determine together with you, in order to match your capacities.

Furthermore, we actively monitor content on mirrors, so that we redirect only to files which actually exists on them. rsync is the most efficient way to do this; scanning through 300.000 files might take only a few minutes with it. The second best method, if rsync is not available, is via FTP, but it is much less efficient (takes considerably longer and places more load on your server). As last resort, we can fall back to HTTP, if neither rsync nor FTP is available. But it crawls. Thus, please do consider adding an rsync module for opensuse content, which allows for much faster scanning of your server.

You may also want to add your mirror to our official mirror list for the released versions or for the development builds, but those lists are not used for the download redirector and might be phased out later. These are wiki pages, simply hit the "Edit" button at the top ;)

Staying informed

The mirror@opensuse.org mailing list (previously called suse-mirror@suse.de) is low-traffic and used mainly for announcements. It is also a suitable place for discussions around mirroring openSUSE content, should the need arise. To subscribe, please write to mailto:mirror+subscribe@opensuse.org and, since it is a closed list, also send a note to mailto:ftpadmin@suse.de and ask to be added.

The general contact address is: mailto:ftpadmin@suse.de

There is an IRC channel named #opensuse-mirrors at freenode.net.

How to set up a mirror

See here for a howto: Mirror_Setup_Howto

rsync modules

The rsync modules on stage.opensuse.org and rsync.opensuse.org are mostly identical. The former has additional content which is yet to be released, and since the latter syncs from it, there is a short sync time lag between them.

Sizes of the rsync modules are triangulated nightly:

• rsync modules of stage.opensuse.org
• rsync modules of rsync.opensuse.org

An example of a commandline syncing from a module could look like this:

 rsync -rlpt rsync.opensuse.org::opensuse-hotstuff-160gb /srv/pub/opensuse/ --delete-after -hi --stats

modules of main interest:

opensuse-hotstuff-160gb: The most requested files, which fit into 160 GB. This currently includes the install repo and CD/DVD media of the latest product, its updates, and the most popular other repositories. This is the most suitable module for mirrors with limited disk space. The majority of requests goes on exactly these files.

opensuse-hotstuff-80gb: An even more restricted selection of most popular files, restricted to 80 GB of space. Use this if your mirror has very limited disk space. Still, the majority of requests goes on the files included in this module, so it is highly useful to mirror "only" these files.

opensuse-updates: This rsync module provides the /update tree, with official updates for released openSUSE distributions, starting with openSUSE 10.3. (To mirror the updates for older releases, check rsync://ftp.suse.com/suse/update/.)

opensuse-full: This rsync module provides the complete content of download.opensuse.org, except the SL-OSS-factory directory. The reason to exclude this directory is the high frequency of updates inside. To mirror the SL-OSS-factory directory, we recommend using drpmsync to fetch this directory instead, it decreases the traffic to less than 10% compared to rsync.

opensuse-full-with-factory: The same as the previous one including the SL-OSS-factory directory containing the Factory Distribution. Again, we do not recommend using this module.

opensuse-source: This rsync module provides the /source tree, which contains source packages of openSUSE 11.1 onwards. Only available on stage.opensuse.org, but without access restrictions.

opensuse-debug: This rsync module provides the /debug tree, which contains source packages of openSUSE 11.1 onwards, and includes released updates. Only available on stage.opensuse.org, but without access restrictions.

modules for mirroring the Build Service repositories:

buildservice-repos: The complete content

buildservice-repos-main: Everything, but not the home: projects of the users

Updates do happen all the time, whenever a repository from the Build Service got rebuilt and updated. It is also possible to get the updates pushed.

modules for mirroring the drpmsync tree:

opensuse-drpmsync: distribution/SL-OSS-factory/drpmsync tree only (OSS part)

opensuse-drpmsync-nonoss: distribution/SL-Factory-non-oss/drpmsync tree only (non-OSS part)

opensuse-full-with-factory-drpmsync: The same as opensuse-full-with-factory, including the drpmsync tree.

Use only if you intend to set up your own drpmsync server.

If you want to mirror these trees, you need to be aware that the enormous number of files (~200000) will cause a considerable load on our rsync server. Please avoid it if possible.

drpmsync.opensuse.org

drpmsync is a sync services for the Factory Distribution. It does reduce the transfer heavily by transmitting only xdelta data, if the local rpm is not older than 1 month.

drpmsync provides access to the SL-OSS-factory directory only. This directory contains always the latest build of the distribution. This means it can also contain an inconsistent or very broken state. It is useful for developers to fetch the latest code or for testers to validate the latest version.

A drpmsync client can get found in the make deltarpm package from http://download.opensuse.org/tools/drpmsync/ and it can be called with this line for example:

  drpmsync drpmsync.opensuse.org:8888/Factory /my/directory

Please note that drpmsync.opensuse.org might be under heavy load, we do look for public mirrors who want also to run a drpmsync server. Please contact us, if you do want to run one as well.

Pushing support for Build Service updates

stage.opensuse.org does also host all content from the Build Service. Since the updates do happen all the time, whenever a new package set got built it is also possible to get the content pushed, instead of polling for it. The obviously requires rsync write access for stage.opensuse.org on your server. The advantages of that method are that

• the mirror is almost always up to date,
• no need to run rsync calls via all repositories. The pushing will only update the repositories which have changed. This does reduce the IO load of the mirror a lot.
• the redirector running at http://download.opensuse.org/ is aware that the packages got updated and can immediately redirect to the mirror.

How to become a pushed mirror?

The usual way (but we can also support a different way) is to open a rsync module on your server, where stage.opensuse.org gets write access. A login and password is optionally possible, but not really needed. Please write a mail to mailto:ftpadmin@suse.de afterwards where you tell us:

• the server name where to push
• the rsync module to be used
• optional account and password
• What you want to mirror (everything or only some certain projects)
• What the public download url will be.
• Any kind of a special wish :)

Then, we need rsync read access to scan your mirror for our download redirector. The download redirector database needs to be updated periodically so it reflects the actual files on your mirror. The scanning happens from 195.135.220.0/22.

Planning: Sizes & Update Frequency

Size	Path of the subtree	Frequency of update
20G	distribution/10.3/iso	rarely/never
51G	distribution/10.3/repo	rarely/never
21G	distribution/11.0/iso	rarely/never
42G	distribution/11.0/repo	rarely/never
947M	distribution/11.1/delta	rarely/never
19G	distribution/11.1/iso	rarely/never
14G	distribution/11.1/repo	rarely/never
25G	update/10.3	frequently (sync it every 4-24 hours)
13G	update/11.0	frequently (sync it every 4-24 hours)
4.6G	update/11.1	frequently (sync it every 4-24 hours) The update trees will continue to grow.
18G	factory	routinely
228G	repositories (including home:/)	very frequently -- push rsync recommended
120G	repositories (without home:/)	very frequently -- push rsync recommended
<160G	[opensuse-hotstuff-160gb]	varies (sync it every 6 hours)
<80G	[opensuse-hotstuff-80gb]	varies (sync it every 6 hours)

(as of March 14, 2008)