Created attachment 14000 [details]
small configuration

Created attachment 14001 [details]
medium configuration

Another thing to check is whether the card for remote management (e.g. the iDrac enterprise) requires to pay some license for accessing to full functionality its software, or we might use standard software without any extra license. In that case we need to know whether the costs of the licenses are included or need to be bought apart, so lists extra the costs.

For the first machine needed, duvel, CPUs are mostly used to compress repository metadata and overall performance is currently mostly limited by storage rather than CPUs (running rpmlint on large packages is currently one of the slow things).

This will change after we get modern SSDs but I don't expect having advanced CPUs to be that useful, having enough cores to do run rpmlint on several packages in parallel or to update repository metadata for all architectures in parallel after we switch to SSDs for example would be what matters whatever the exact CPU is.

Regarding memory, if the price is reasonable it is probably better to go with more (like 4x32) from the beginning as our main constraint is having someone to travel to the data center and historically it has been a (at least) year long project to upgrade a server.

Regarding build machines, that's the part where using VMs is easier as we don't need large storage, just fast storage, many cores/threads and more than 1G ram per thread. And merging ram + switching to SSD (even a single small one like 120G) on some machine we already have would probably go a long way.

The oracle arm build VMs which we were using to build very quickly 3 packages at once had 150 GB storage, 24 cores and 36GB ram.
Our current build machines, ecosse has 24 threads (2 * "Intel(R) Xeon(R) CPU E5-2640 0 @ 2.50GHz") and 4*8GB ram with 20 empty slots, rabbit has 12 threads (Intel(R) Xeon(R) CPU E5-2430 0 @ 2.20GHz) and 3*16GB.
I am not sure it is worth keeping the current rabbit given its CPU but we should add more ram if we do, else we can transfer its ram to ecosse. If ecosse gets 3*16+4*8G=80G ram + a small ssd, it should do well.

CC: (none) => pterjan

(In reply to Pascal Terjan from comment #4)

> For the first machine needed, duvel, CPUs are mostly used to compress
> repository metadata and overall performance is currently mostly limited by
> storage rather than CPUs (running rpmlint on large packages is currently one
> of the slow things).
> 
> This will change after we get modern SSDs but I don't expect having advanced
> CPUs to be that useful, having enough cores to do run rpmlint on several
> packages in parallel or to update repository metadata for all architectures
> in parallel after we switch to SSDs for example would be what matters
> whatever the exact CPU is.
> 

beside faster SSDs which seems to be a must (we need them in the enterprise class), but would have sense to use a huge ramdisk for scratch for running the rpmlint and metadata updates?

E.g. if we suppose to have a 256GB RAM, and then using 64GB for scratch:

mount -t tmpfs -o size=65536M tmpfs /mnt/ramdisk

and then validating the metadata on the ramdisk scratch rather than directly on the SSDs (that would also preserve SSDs writing cycles). ramdisk should have faster access than any SSDs. Would you able to bench the gain/difference ramdisk vs plain SSDs? It could be interesting in the case for "duvel2", e.g. we might choose CPUs of less power but with more extra RAM.

@Lewis Smith i add you as you are interesting on this

CC: (none) => j.alberto.vc, lewyssmith

(In reply to Pascal Terjan from comment #4)

> For the first machine needed, duvel, CPUs are mostly used to compress
> repository metadata and overall performance is currently mostly limited by
> storage rather than CPUs (running rpmlint on large packages is currently one
> of the slow things).
> 
> This will change after we get modern SSDs but I don't expect having advanced
> CPUs to be that useful, having enough cores to do run rpmlint on several
> packages in parallel or to update repository metadata for all architectures
> in parallel after we switch to SSDs for example would be what matters
> whatever the exact CPU is.
> 
> Regarding memory, if the price is reasonable it is probably better to go
> with more (like 4x32) from the beginning as our main constraint is having
> someone to travel to the data center and historically it has been a (at
> least) year long project to upgrade a server.
> 
> Regarding build machines, that's the part where using VMs is easier as we
> don't need large storage, just fast storage, many cores/threads and more
> than 1G ram per thread. And merging ram + switching to SSD (even a single
> small one like 120G) on some machine we already have would probably go a
> long way.
>
 
Regarding new duvel and new sucuk, according to the initial config, they are not intended as X86_64 building hosts, so shouldn't we add some new extra building host too to the list? Dual Xeon E5 2640 has nowadays less power than a desktop entry level with a single Ryzen 5600G.

last but not least, can't we use the new duvel for adding a tiny virtual machine with KVM/libvirt for madb too? Then that can be easily moved across hosts.

So looking at this, if we go with a used option, this Dell 730[1] seems like a good option. It uses the Xeon E5 v4, dual socket DDR4, so we can happily go for a very wide build queue with a lot of memory. There are supermicro and HP options, not sure what the management system for the supermirco servers is like, but the HP one comes with a fee, seems the Dell has it included.

While funds seem available to cover a newer server, going with a Xeon scalable in a Dell 740 or equivalent seems like a poor option, and going with a new AMD server would be a better option in that case. 
If NVME drives are really needed, expansion cards are available [2][3]

This [4] build seems to meet what Thomas and others have asked for, shipping to France is possible, €3600 + tax, which I assume we are exempt from. Including shipping. This can be optimised for more memory and storage is required, switching to a 2.5 and 3.5 disk split helps with that.


Main questions around this are the SSD loadout as the cost/Gb for the 800Gb drives is very high, going with fewer drives at 3.84Gb would make a lot more sense from a size perspective, would losing that redundancy be an issue?

[1] https://www.bargainhardware.co.uk/dell-poweredge-r730xd-26-sff-configure-to-order
[2] https://www.bargainhardware.co.uk/pcie-3-0x4-to-m-2-nvme-converter-fh-lp-new
[3] https://www.bargainhardware.co.uk/dell-gy1td-poweredge-u2-nvme-enablement-card-lp-pcie-x16-0gy1td
[4] https://www.bargainhardware.co.uk/dell-poweredge-r730-xd-2u-24x-2-5-sff-configure-to-order?c_b=60171

Quote direct from Dell for the AMD server proposed by Thomas. €19500, however, it seems that a far better price is available if we go through a reseller/talk to someone at the company.

Would be good to get some input from the accounts to see how flush Mageia is atm.

https://www.dell.com/fr-fr/shop/pdr/poweredge-r7515/per751501a?selectionState=eyJPQyI6InBlcjc1MTUwMWEiLCJNb2RzIjpbeyJJZCI6MTUwMiwiT3B0cyI6W3siSWQiOiJHQkVaV084In1dfSx7IklkIjoxNTAzLCJPcHRzIjpbeyJJZCI6IkdLSlg2RDkifV19LHsiSWQiOjE1MTAsIk9wdHMiOlt7IklkIjoiRzdOMUQ2WCJ9XX0seyJJZCI6MTUxNCwiT3B0cyI6W3siSWQiOiJHODNOQzVCIn1dfSx7IklkIjoxNTMwLCJPcHRzIjpbeyJJZCI6Ikc0T0w5NkgifV19LHsiSWQiOjE1MzEsIk9wdHMiOlt7IklkIjoiRzc1MlFFQiJ9XX0seyJJZCI6MTUzNCwiT3B0cyI6W3siSWQiOiJHU0ZURzRZIn1dfSx7IklkIjoxNTQxLCJPcHRzIjpbeyJJZCI6Ikc3WkMwT0gifV19LHsiSWQiOjE1NjAsIk9wdHMiOlt7IklkIjoiR0ZSQzlFTCIsIlF0eSI6NH1dfSx7IklkIjoxNTcwLCJPcHRzIjpbeyJJZCI6IkcwUFlVUTkiLCJRdHkiOjN9LHsiSWQiOiJHSlVPSDlZIiwiUXR5Ijo2fSx7IklkIjoiRzNaSk0wSyJ9LHsiSWQiOiJHM1NEWFIyIiwiUXR5Ijo4fV19LHsiSWQiOjE2MDAsIk9wdHMiOlt7IklkIjoiRzVaRjdLVSJ9XX0seyJJZCI6MTYyMCwiT3B0cyI6W3siSWQiOiJHRFNDSlc1In1dfSx7IklkIjoxNjIxLCJPcHRzIjpbeyJJZCI6IjUxODA1MSIsIlF0eSI6Mn1dfV0sIlRpIjoiIiwiRGkiOiIifQ%3D%3D&cartItemId=

(In reply to Donald from comment #9)

> Quote direct from Dell for the AMD server proposed by Thomas. €19500,
> however, it seems that a far better price is available if we go through a
> reseller/talk to someone at the company.
> 
> Would be good to get some input from the accounts to see how flush Mageia is
> atm.
> 
> https://www.dell.com/fr-fr/shop/pdr/poweredge-r7515/
> per751501a?selectionState=eyJPQyI6InBlcjc1MTUwMWEiLCJNb2RzIjpbeyJJZCI6MTUwMiw
> iT3B0cyI6W3siSWQiOiJHQkVaV084In1dfSx7IklkIjoxNTAzLCJPcHRzIjpbeyJJZCI6IkdLSlg2
> RDkifV19LHsiSWQiOjE1MTAsIk9wdHMiOlt7IklkIjoiRzdOMUQ2WCJ9XX0seyJJZCI6MTUxNCwiT
> 3B0cyI6W3siSWQiOiJHODNOQzVCIn1dfSx7IklkIjoxNTMwLCJPcHRzIjpbeyJJZCI6Ikc0T0w5Nk
> gifV19LHsiSWQiOjE1MzEsIk9wdHMiOlt7IklkIjoiRzc1MlFFQiJ9XX0seyJJZCI6MTUzNCwiT3B
> 0cyI6W3siSWQiOiJHU0ZURzRZIn1dfSx7IklkIjoxNTQxLCJPcHRzIjpbeyJJZCI6Ikc3WkMwT0gi
> fV19LHsiSWQiOjE1NjAsIk9wdHMiOlt7IklkIjoiR0ZSQzlFTCIsIlF0eSI6NH1dfSx7IklkIjoxN
> TcwLCJPcHRzIjpbeyJJZCI6IkcwUFlVUTkiLCJRdHkiOjN9LHsiSWQiOiJHSlVPSDlZIiwiUXR5Ij
> o2fSx7IklkIjoiRzNaSk0wSyJ9LHsiSWQiOiJHM1NEWFIyIiwiUXR5Ijo4fV19LHsiSWQiOjE2MDA
> sIk9wdHMiOlt7IklkIjoiRzVaRjdLVSJ9XX0seyJJZCI6MTYyMCwiT3B0cyI6W3siSWQiOiJHRFND
> Slc1In1dfSx7IklkIjoxNjIxLCJPcHRzIjpbeyJJZCI6IjUxODA1MSIsIlF0eSI6Mn1dfV0sIlRpI
> joiIiwiRGkiOiIifQ%3D%3D&cartItemId=

That's close to the 18000E:

https://www.dell.com/fr-fr/shop/nos-solutions-serveurs-stockage-et-r%C3%A9seaux/smart-selection-poweredge-r7515-serveur-rack/spd/poweredge-r7515/per751501a?configurationid=e386aa6f-9824-4026-8471-fb65e9da4560

that I posted yesterday in ML.

We need to now if we are close to the budged possibilities, because even on the hypothesis of 40% discount, you get 22-23000E for 2 servers, so if we exceed by 10x times that not viable.

Yesterday I tried to click on the contact button to chat with an advisor in the Dell fr site to get a commercial contact, I don't speak/write french, so had to arrange with translator. What he ask for getting a quote is a:

"J'aurai également besoin d'un nom de société, d'une adresse de livraison et de facturation, d'un nom de contact ainsi que d'un numéro de téléphone où vous êtes joignables et le numéro de Siret . (Ces informations figureront au niveau de l'entête de votre devis)"

which is

"I will also need a company name, a delivery and billing address, a contact name as well as a telephone number where you can be reached and the Siret number. (This information will appear in the header of your quote)"

I don't have such data. I left my email address, asking to get a contact from a commercial, but at the moment I haven't received yet anything. Maybe some french native speaker can try too.

(In reply to Donald from comment #8)
> So looking at this, if we go with a used option, this Dell 730[1] seems like
> a good option. It uses the Xeon E5 v4, dual socket DDR4, so we can happily
> go for a very wide build queue with a lot of memory. There are supermicro
> and HP options, not sure what the management system for the supermirco
> servers is like, but the HP one comes with a fee, seems the Dell has it
> included.
> 
> While funds seem available to cover a newer server, going with a Xeon
> scalable in a Dell 740 or equivalent seems like a poor option, and going
> with a new AMD server would be a better option in that case. 
> If NVME drives are really needed, expansion cards are available [2][3]
> 
> This [4] build seems to meet what Thomas and others have asked for, shipping
> to France is possible, €3600 + tax, which I assume we are exempt from.
> Including shipping. This can be optimised for more memory and storage is
> required, switching to a 2.5 and 3.5 disk split helps with that.
> 
> 
> Main questions around this are the SSD loadout as the cost/Gb for the 800Gb
> drives is very high, going with fewer drives at 3.84Gb would make a lot more
> sense from a size perspective, would losing that redundancy be an issue?

Beyond the 3 SAS SSD disks for the system, in RAID1, if you are referring to the 6 SSD 1.92TB in RAID6, that should use 4 disks for RAID6 (thus allowing 2 disk failure) with a total, as you said, of 3.84 TB and 2 disks for spare about that. IMHO less redundancy means someone have to go to the datacenter to change the disks more often than with 1 disk only.

For a cheaper solution he suggested 2 arrays of 8 disks mechanical (so not SSD), each of 1.2TB, with RAID 6 and 2 disks spare each. So a total of 2*4.8TB array.

(In reply to Donald from comment #8)

> So looking at this, if we go with a used option, this Dell 730[1] seems like
> a good option. It uses the Xeon E5 v4, dual socket DDR4, so we can happily
> go for a very wide build queue with a lot of memory. There are supermicro
> and HP options, not sure what the management system for the supermirco
> servers is like, but the HP one comes with a fee, seems the Dell has it
> included.

Supermicro remote management usually uses their how BMC card with custom software which should be IPMI compatible.

Interesting some used server starting from 50-100E. Dunno about how the hosting is based on marseille, how many servers can be added in the rack, etc., but we might got an extra used 100+100E server maybe with 32GB RAM to be used for micro VMs like those required for madb.

https://www.etb-tech.com/dell-poweredge-r7515-configure-to-order-btor7515.html

This comes out to £7000 for a very similar configuration to the one from Dell official above but with 24 core Epyc and 128Gb ram. Its only 25 more than the 16 core. 3 year warranty but would need a little extra for the additional sfp card.

(In reply to Donald from comment #13)
> https://www.etb-tech.com/dell-poweredge-r7515-configure-to-order-btor7515.
> html
> 
> This comes out to £7000 for a very similar configuration to the one from
> Dell official above but with 24 core Epyc and 128Gb ram. Its only 25 more
> than the 16 core. 3 year warranty but would need a little extra for the
> additional sfp card.

I tried your site for the "medium configuration" and got 7500€ Tax Excluded (and without the SFP card), with CPU EPYC 7352 24 core (it's a Zen2, late 2019, core Rome), performance similar to the initial 7313P (Zen3, core Milan, early 2021), with 64GB RAM (2x32). For 7700€ (Tax excluded) you might get the EPYC 7452 CPU (Zen2, core Rome, late 2019), which is 32core and a bit faster. Extra +64GB RAM is further 300E.

Apparently they deliver in the EU, and VAT seems due at custom. VAT in France ashould be 20%, it raises to 9000-9500€ (https://www.etb-tech.com/dell-poweredge-r7515-configure-to-order-btor7515.html#delivery -> says: "Since 1st January 2021 EU customers will no longer be charged UK VAT by ETB instead local VAT will be collected by the courier on behalf the relevant authorities in your country. There will also be a small customs processing fee payable to the courier upon delivery of your order.").

CC: (none) => mageia

We have enough options here, need input from Council on what they are willing to spend. Will ask there

I believe it is worth spending so we have a fast system and it will be more fun to use, not having to wait so much.

CC: (none) => fri

I'm sorry to sound negative, but I think we are not taking that issue by the right side.

First we need to define what we need wrt to HW configuration, instead of focussing on a specific HW vendor part number set.

Do we need:

- 3 x 10Gbits Eth cards or 2 x 1 Gbit or ... TBD That should take in account external connectivity, the type of ports (Eth 1/10/20/50/100 Gb, form factor of the ports) the switch of our provider can give us, and then if we want bonding or not for redundancy...
- HW Raid or SW Raid or no Raid, then the size of the storage 10 TB ? 20 TB ? then the type of HW techno HDD/SSD (the last one being IMHO preferred these days) ... TBD
- How many cores do we need to handle parallel processing 32 ? 48 ? which could be in 1, 2 or 4 physical procs ? Preferring less Ghz for more cores (which I think is better including from a price point reason) or higher frequency for less cores ? ... TBD
- How much RAM : 64, 128 GB ? how many DIMMs ? frequency of the DIMMs, ... this is probably by far what will give us more perf: for the same budget we should prefer more ram and less GHz typically.
- How many power supplies ? What quality ?
- Do we need expansion capabilities (I'd say no if we configure the machines correctly) ?

And that list should be done per machine, knowing exactly their destination. Which means that first we need to look at what we currently have wrt these caracteristics (inventory first) and decide based on the workload in the future.

Please stop talking IPMI in the 21st century. Now servers are proving Redfish access through their BMC (HPE, Dell, and many others), and that's what we should have to get all latest services available (which means another port !!

Then we need to know whether we want to buy a machine with a rather fixed configuration or not, knowing that adding stuff in years from now is very expensive and generally not the right approch (I tend to buy correctly configured systems for 5/10 years and not have them change, also here because it's far away)

Then we can configure that proposal of conf using the various HW providers we trust can make the job. I think we need to create a spreadsheet with the current info on existing systems, then the wished info on the future systems and then only starting thinking to ordering.

We *need* for legal reasons to have a french delivery, a french invoice is probably way easier, so a french person

And I'm volunteering to bring the HW to the DC in Marseille and perform the installation on site.

My background: I've been one of the few HPE WW consultant for Linux on HPE servers for the last 25+ years and should be able to help, even if another manufacturer is chosen ;-) And I've been advising hundreds of HPE customers to build their Linux infra during that period.

(In reply to Bruno Cornec from comment #17)

> I'm sorry to sound negative, but I think we are not taking that issue by the
> right side.
> 
> First we need to define what we need wrt to HW configuration, instead of
> focussing on a specific HW vendor part number set.
> 
> Do we need:
> 
> - 3 x 10Gbits Eth cards or 2 x 1 Gbit or ... TBD That should take in account
> external connectivity, the type of ports (Eth 1/10/20/50/100 Gb, form factor
> of the ports) the switch of our provider can give us, and then if we want
> bonding or not for redundancy...
> - HW Raid or SW Raid or no Raid, then the size of the storage 10 TB ? 20 TB
> ? then the type of HW techno HDD/SSD (the last one being IMHO preferred
> these days) ... TBD
> - How many cores do we need to handle parallel processing 32 ? 48 ? which
> could be in 1, 2 or 4 physical procs ? Preferring less Ghz for more cores
> (which I think is better including from a price point reason) or higher
> frequency for less cores ? ... TBD
> - How much RAM : 64, 128 GB ? how many DIMMs ? frequency of the DIMMs, ...
> this is probably by far what will give us more perf: for the same budget we
> should prefer more ram and less GHz typically.
> - How many power supplies ? What quality ?
> - Do we need expansion capabilities (I'd say no if we configure the machines
> correctly) ?
> 
> And that list should be done per machine, knowing exactly their destination.
> Which means that first we need to look at what we currently have wrt these
> caracteristics (inventory first) and decide based on the workload in the
> future.

The initial proposed configuration was specified in the attached files as starting point, which were the initial configuration prepared by Thomas. I think he was knowing the kind of tasks the servers were used for. Just remove the brand and you have the amount of memory, which is 64GB. dual HotPlug Redundant Power Supply, Hardware RAID (RAID6) (so no sw RAID, no sw network bonding). For the "power supply quality" what do you mean exactly?

For top hardware configuration like 100Gbit networking etc., it was asked initially in mailing list and there was no such needing.

According to the initial mail, those are for replacing sucuk and duvel, which are not building servers, i.e. they are not used for compiling packages, though one of them is used for "compressing repository metadatas" (see comment on https://bugs.mageia.org/show_bug.cgi?id=32306#c4). And for warranty initially was choosen "5 years standard (no pro) support". We might add also further powerful building nodes, but later.

Also for things like number of physical CPUs, core, etc, we can't ignore the CPU generation. Even thinking to keep the hardware for the next 5 or 10 years probably the last CPU generation (i.e. late 2023) and with DDR5 is not yet worthwhile at the moment for cost/beneficts ratio.

The current "inventory" for servers are listed here, as already reported. That's all what I was able to collect:

https://wiki.mageia.org/en/Marseille_Servers

If someone has more datas, feel free to improve that wiki page (and also reporting here).

E.g. duvel is a dual Xeon E5-2630v2 with 32GB DDR3-1600 RAM, which is a CPU from ten years ago. It was top hardware at that time, but it's clear that any 500E PC that we might use today as home-theater PC might be more powerful than that at any task, but of course it has not the same redundancy and reliability.

For the memory, number of physical CPUs etc., DIMMs, what you said is true, but these are kind of a 2nd level optimizations to the initial survey. Your approach is more close to corporate manager, and requires precise and exact data and precise benchmarking about CPU and IO's load, but at the moment we don't have (and probably we wouldn't) such precise data, so we need to proceed by successive approximations.

Also if we stuck the configuration to the current exact tasks we are precluding the possibility for the software infra to grow to something newer, so I think there should be to keep some margin to the configurations.

IMHO over bi-processor motheboard (so more than 2 physical CPU) they are way more expensive, so I think, at most a biprocessor.

E.g. we might decide what would be the best hardware optimized for "compressing repo metadata". Just an example: an extra 128 or 256GB RAM (even at a lower frequency) to be used as a tmpfs/ramdrive can be faster than a RAID0 NVMe (and you haven't even the endurance problem). But if you can't sustain that from economic point of view, then you have to get the plain disks, first with SSD and then dropping to HDDs. Or maybe you find that even with best hardware you get only a 15% increase in performance.

The problem here is that we are stalled on the costs, not on the config, because we are not knowing what can be affordable and what is not from costs point of view. In the first survey we arrived at 18-30000 euros for two servers. Even if we cut in two, suppose a 50% discount, you get 9-15000 euros. If we can't afford such amount, because for instance we can't go beyond 3-4000E for both the servers, then we should rethink the configurations and other ways, including the possibility to get, as suggested, used hardware (why not?) if that satisfies the needing.

> 
> Please stop talking IPMI in the 21st century. Now servers are proving
> Redfish access through their BMC (HPE, Dell, and many others), and that's
> what we should have to get all latest services available (which means
> another port !!

IPMI was just a generic synomim jargon for an *hardware* card intended for remote management, where you can  have basic control of bios, boot, power reset; it's clear that every vendor has it's own one (Supermicro has its own, iDRAC for Dell, iLO for HP, etc.). I might have used improperly, sorry for the confusion.

> 
> Then we need to know whether we want to buy a machine with a rather fixed
> configuration or not, knowing that adding stuff in years from now is very
> expensive and generally not the right approch (I tend to buy correctly
> configured systems for 5/10 years and not have them change, also here
> because it's far away)

ok.

> 
> Then we can configure that proposal of conf using the various HW providers
> we trust can make the job. I think we need to create a spreadsheet with the
> current info on existing systems, then the wished info on the future systems
> and then only starting thinking to ordering.

Feel free to improve the configuration moving the initial configuration in a spreadsheet (and post in attach).

> 
> We *need* for legal reasons to have a french delivery, a french invoice is
> probably way easier, so a french person

which exact legal reasons? Why it's easier, we shouldn't exclude other possibilities a priori. The main issue I see is probably with warranty and with hardware onsite support over the years. If you buy for instance in UK, then it's more difficult to have an intervention over Marseille.

> 
> And I'm volunteering to bring the HW to the DC in Marseille and perform the
> installation on site.

Great.

Also if you want to do an prepare a configuration on the HPE side then feel free to post the configuration/costs.

BTW, are that possibilities to get free HW from some sponsor?

Regarding Marseille, after the recent out of service that keepen MLs stopped for one week, we might also re-think to some tiny fail-over infrastructure in another DC.

Hello, I don't think that buying in any country from EU is a problem. But from GB, it is, because it is no more in the EU.

CC: (none) => yvesbrungard

(In reply to papoteur from comment #19)
> Hello, I don't think that buying in any country from EU is a problem. But
> from GB, it is, because it is no more in the EU.

It's a bit more complicate because of customs (douanes), see bottom of comment #14, though feasible. But in any case it depends on what it can be saved globally on the configuration to see if that can be worthwhile (e.g. in case of used hardware).

any progress

I've never built rack servers but looking at the current servers, the two slowest are only running on 12 threads each. Would it not be possible to just buy a AMD AM5 motherboard and put a 16 core (32 thread) CPU in it? You could even find server grade AM5 boards if you look around a bit.

CC: (none) => xerxes2

Just another thought about this discussion:

maybe just renting some hardware at a hoster (asking for discount) can be an option too:
https://www.hetzner.com/dedicated-rootserver/matrix-ax/
Upgrading is easier, as you don't have to worry about the "old hardware" and just rent newer.

Even, renting 2 or more nodes with medium size can be more cost effective than to have one big new one. If a build just use one core the bigger one is even slower than to have 2 or more nodes in parallel for the same cost.

You can find some good used hardware here too:
https://www.hetzner.com/sb/
which performs quite good for the price you pay. I guess other hosters may have similar offers. Maybe rethink if we really have to buy the hardware ourself.

CC: (none) => mageia

AIUI, we get free colo space, so if we keep our hardware for 5 years, it's cheaper to go with buying our own if it costs us less than €6240 (compared to that medium-sized Hetzner option). Compared to the higher-end Hetzner machines (which is closer to what we were looking at, I think), we have €12000 to spend to break even.

CC: (none) => dan

For the server "size", consider that actually the servers are 4. For the infra twos keep up the systems (so NFS, repo, WEB, mailing lists, package signing, etc.), mostly, if I did the math correctly, around 8TB in RAID of storage (and that the size needs to be increased so to not navigate always at 95% of filesystem size). The other 2 are building nodes. 

This for x86_64/x86. And possibly with a remote management card so you can install whatever OS you want, etc.

The priority is to upgrade first the infra servers which keep up the infrastructure, then later (or at same time) add further building nodes. Building nodes can be as many as we can and that the dataroom could afford (power availability, racks, etc.), and even planning of adding extra nodes as used hardware. Infra servers also communicate with nodes through internal ethernet (or fiber) cards.

In theory the services (MLs, Web, etc.) could also be splitten even further across further servers.

First to say, I am not connected to hetzner, just using them as a flexible hoster for my company, as I can run mageia there and can freely configure vlan for my infrastructure there.

Renting the hardware can have a few benefits:
a) you don't need the whole money at the beginning
b) you can switch to newer hardware every time you want and have the time for the migration, or give them a demand to mount the old disks into newer hardware
c) you don't (really) have to worry about hardware defects like disks
d) you can also rent some very cheap virtual hosts to split up some services, which must not run on the main infrastrucure (e.g. mail, website), or use them as standby built environment (in case of a rebuilt, just start 10 more hosts and let them built in parallel and remove them after our rebuilt is finished)
e) maybe get a sponsorship for the rent of a server. At the moment I find it hard to make donations to mageia as a company.

I am not familiar with our build environment, but to my experience, it is sometimes better to have (e.g.) 4 cheaper build nodes, which have their own io, ram, disk instead of having one very expensive one. Some taks like linking run on just one core and may leave all others idle, because of the build schedule.


I just want to make a few suggestions, it might make sense not to buy the own equipment. But in the end, it is the decission of council/sysadmin.

Thank you for interests and suggestions. The infos I added were to allow to eventually define a better configuration on the rent site so to have a more realistic ideas of the break even costs.

BTW, for the root servers has anyone had a past experience with such kind of services?

Although I am on the CC list, I have not received any posts for this bug; which explains my ignorance whenevr I comment on this matter. Never mind.

A money thought: we are rumoured to have some Bitcoins. If we could sell these, we would not be cramped for funds... I search in vain for someone who has actually dealt in them. It looks complicated.

The Bitcoins were brought up early in the discussion, somewhere, and rejected. I don't remember details, but it had something to do with French rules concerning cryptocurrency and taxes. As I recall, it was something about the person that signs the conversion to money has to declare it as income; a non-profit organization like Mageia can't do it.

I may have that completely wrong, so don't rely on that without checking it out. Remember, I have never been French, nor have I ever been an international financial expert.

(In reply to Lewis Smith from comment #28)
> Although I am on the CC list, I have not received any posts for this bug;
> which explains my ignorance whenevr I comment on this matter. Never mind.
> 
> A money thought: we are rumoured to have some Bitcoins. If we could sell
> these, we would not be cramped for funds... I search in vain for someone who
> has actually dealt in them. It looks complicated.


We have no acces to this account. So to summarize: We have 0 bitcoins and 0 active bitcoins account.

So I had been contacted back by a french person from servermall today to discuss the configuration and our needs. I explained the context, they can provide a discount if we buy multiple servers. Their stock is in Lituania, but they deliver for free in France, I I could get and test everything before driving with the machines to Marseille and perform the setup. They also provide 5 years warranty.
https://servermall.com/fr/sets/hp-rack-server/

I got that quote today for a duvel-II type of system:
DL380 Gen10 12LFF Refurbished
2* Gold 6126 (12C, 24T, 2,60Ghz, 125W)
4*32GB DDR4 2666y
HPE P408i-a 2GB Gen10
ILO5 Standard
FLR 2x10GB RJ45
2*HPE 800W Power Supply
2* SSD 1.92TB SATA 2,5+ Tray Caddy 3.5 (NEW)
6*HDD 12TB SATA 3,5 + Tray Caddy 3.5 (NEW)
2*Adapter 2.5-3.5
Without OS
Rail Kit 19’
5 years warranty from Servermall

Price: 5783€ no VAT. If we do have a VAT number we can avoid to pay for
it as I was explained. Nicolas/Pascal/Maat any clue ? (From my readings,
I think we can't avoid to pay it in France which is 20%).

So Price: 6939.6€ VAT incl.

So I could ask for 2 larger SSDs to reach the "dreamed" conf Giuseppe
and I are discussing here. And that should be a good one. (should be
around 600€ more). It's 12 LFF so there is even room for adding disks if
needed (for raid6 conf or more space)

The guys only do Intel based plaform, so no AMD available as
refurbished.

So looking at the price for such a machine, I think we can order 4
servers duvel-II, sucuk-II, ecosse-II and rabbit-II and stay in the
budget mentioned by council.

I wonder whether sucuk-II could be old-duvel once moved ? Would spare
some money. We would just have to buy additional memory for it (moving
from 32GB to 128GB).
The 8 GB DIMM (we have 4) costs 21€ no-VAT
(https://www.renewtech.fr/dell-hmt41gr7afr8a-pb.html)
Buying 12 of these costs 302€ VAT and bring old-duvel to 128GB. lshw shows
that there is enough room to host them (24 DIMMs slots available and 4
used so 20 free and 2 CPUs).


Then we can focus on adding 2 new x86_64 build nodes. Then from
servermall I think we could target these:

HPE DL360 Gen10 8SFF Refurbished
2 × Intel Xeon Gold 6126 (12C 19.25M Cache 2.60 GHz)
4 × 64GB DDR4 RDIMM
RAID HPE P408i-a (2GB+FBWC)
iLO 5 Standard
2x Power supply HP 500w
2 ports 10GB Base-T RJ-45 PCI-e
4 port 1Gb (Integrated)
Rack mount kit 19"
2 × SSD 1.92TB SATA 2.5" + Tray Caddy (NEW)
Price €3 397 no-VAT so 4076€ VAT incl.

2 of these would be 8152€


Recap:

duvel-II: 7540€
suckuk-II: 302€
rabbit-II: 4076€
ecosse-II: 4076€

Total: 15994€

So fits in the budget and I think we will have with that a new shiny
infra to manage the distro.

So we could add to this a new backup server (which could be either
physical (8k€ more again in budget) or maybe this time hosted elsewhere with
the appropriate storage - Hetzner typically)

Now the most difficult. I can deal with the discussion with servermall,
get a discount on top of this, get the delivery at my home, setup mageia
on them with an auto_inst so we can easily reproduce, and have them
ready for ssh connection and puppet ready for operation. With iLO5 we should
be able to take control remotely once installed in
Marseille.

Now, I need DC contact to study network connectivity, rack space and
power as the best approach would be to install them alongside the
existing infra. And for me to make later a scond trip to Marseille if
needed to remove what the old-sucuk later on.

I need feedback from you all sysadmin wrt to the proposal so we can make
progresses.