6

I would like to create an RPM repository for Fedora packages on my local network. Due to storage limitations, I want the repository to be empty initially and download packages once they are accessed.

Background

I work a lot with local VMs. Anytime I create a new VM and install Fedora, a lot of packages are downloaded from the internet, and most of the downloaded packages are the same. To speed up the process I would like the RPMs to be cached on a server located on the same network.

Similar questions have been answered with a combination of createrepo & reposync. I do not like the reposync part, because I don't want to clone the whole repository up front when I need only some of the packages.

Ideal Solution

I would like the server on my local network to act as an RPM repository for my Fedora installations. It should pass-through the metadata from whatever is configured in /etc/yum.repo.d/*. The server should deliver the requested RPM if it is present in the local cache, or else download it and then deliver it.

A less ambitious approach would be to configure a single RPM repository instead of https://mirrors.fedoraproject.org/... and just use an http proxy.

Update: 02 Nov. 2015

I already have an nginx running on the network, so I played around with a combination of proxy_pass and proxy_cache. It kinda works, but IMHO it has more drawbacks than benefits:

  • a separate configuration for every repo configured in /etc/yum.repo.d/*.
  • can't use metadata from https://mirrors.fedoraproject.org/ because of alternate mirrors.

I dropped the nginx thing and installed squid, as suggested in comments. squid works great for me. With the store_id_program configuration, I am even able to use the alternate mirrors and still hit the cache, no matter where the RPM came from originally.

G-Man Says 'Reinstate Monica'
  • 22,130
  • 27
  • 68
  • 117
Yevgeniy
  • 161
  • 1
  • 4
  • 3
    The only thought in my head while reading your Q was "use squid" -- and then you said it – Jeff Schaller Oct 31 '15 at 21:08
  • yes, squid is the answer. note that some .rpms are quite large (e.g. kernels can be 30-40MB) so you'll have to increase squid's `maximum_object_size` from default 4MB to, say, 64MB. You might also want a `refresh_pattern .rpm ...` rule to keep rpm files longer than the default. – cas Oct 31 '15 at 23:31
  • btw, you can also use a squid url-rewriter to match the directory pattern of distro rpm repos and rewrite them to use just the one nearby mirror. – cas Oct 31 '15 at 23:33
  • and this LWN article from 2009 is old but still worth reading https://lwn.net/Articles/318658/ – cas Oct 31 '15 at 23:36
  • and, finally, according to http://www.sotechdesign.com.au/how-to-cache-yum-repositories-on-centos-using-apt-cacher-ng-on-debian-or-ubuntu/ you can use Debian's `apt-cacher-ng` to cache yum repositories as well as .deb repos....but you have to do it in a debian vm because it probably isn't ported to fedora/rhel/centos/etc – cas Oct 31 '15 at 23:39
  • thx for your suggestions. right now i playing around with nginx (reverse_proxy + cache), because i already have a nginx running on the server, but squid is the next on my list in case nginx does not work out as expected. – Yevgeniy Nov 01 '15 at 09:12

4 Answers4

4

Here you can find fine-tuned squid.conf for rpm caching:

https://github.com/spacewalkproject/spacewalk/blob/master/proxy/installer/squid.conf

You just should modify memory and port setting.

msuchy
  • 1,408
  • 8
  • 6
  • 1
    thx for sharing your configuration. i surely can pinch something from you. feel free to take a look at my configuration: https://github.com/yevmel/squid-rpm-cache – Yevgeniy Nov 04 '15 at 20:06
2

I'm not sure if you found a good solution for this but I ended up writing a quick little Go program. It acts as a caching proxy and keeps around RPMs, but proxies upstream to database files so it's always correct. It's configured for CentOS, Fedora EPEL, and Arch Linux for now.

You can see it at https://github.com/yobert/remirror

Yobert
  • 139
  • 4
1

apt-cacher-ng works great. We are using it for some times and it works fine. And remember to only use http mirrors (and not https ones) when using it to make it more efficient (naturally, it can't see & cache https traffic):

To configure fedora official repositories to only use http mirrors, you can add &protocol=http at the end of the metalink config url, e.g. for fedora repository it'll become:

metalink=https://mirrors.fedoraproject.org/metalink?repo=fedora-$releasever&arch=$basearch&protocol=http

https://www.unix-ag.uni-kl.de/~bloch/acng/

A caching proxy. Specialized for package files from Linux distributors, primarily for Debian (and Debian based) distributions but not limited to those. See documentation of Apt-Cacher to learn what it's good for.

hedayat
  • 69
  • 4
  • Not sure why I got negative vote, but I've tried and it works fine. Probably the voter assumed that it only works for .deb, but if (s)he've read the link it'd be clear that it works for rpm repos too. – hedayat Dec 07 '18 at 08:47
  • »Attempts to add apt-cacher-ng support ended up in pain and the author lost any motivation in further research on this subject.« https://www.unix-ag.uni-kl.de/~bloch/acng/html/distinstructions.html#hints-fccore It would be very helpful to get detailed instructions (where do I get the package from? it's not in fedora or EPEL it seems) from people who uses this in "production" at the moment. – LiveWireBT Oct 23 '21 at 19:43
  • Currently, we've set up apt-cacher-ng under Ubuntu, but it serves both Ubuntu and Fedora. So, I've no experience in setting up apt-cacher-ng under Fedora, although installing from source should work. But if the settings we use for Fedora is required, I can see and post here. About the author, I've emailed him for better Fedora support but have not heard back. – hedayat Oct 24 '21 at 20:10
0

Yum wiki contain page with review of caching solutions

Some of it on topic:

  • Install "IntelligentMirror", and register with MirrorManager. 

    Pros
    Zero configuration, on the client side.
    Post setup it mostly just works and everything should be immediate.
    Only downloads what is required by the users.
    Fully automated server side, once setup/working. 
Cons
    Requires setting up a local Squid + Apache-httpd + IntelligentMirror to serve the data.
    If the server is down then metalinks/MM will route you to an external mirror.
    Only intelligently caches packages, not metadata.

  • (beta) avahi-packages-server.py and yum avahi support (see: http://james.fedorapeople.org/yum/avahi)

    Pros
    Zero configuration client.
    Zero configuration server.
    Only downloads what is required by the users. 
Cons
    Not upstream yet (still beta).
    A client that doesn't run the avahi server side doesn't share it's downloads (so all clients need to run the "server").
    Requires working avahi on the network.
mmv-ru
  • 605
  • 6
  • 13