Milliways - User contributions [en]

EMF2026Gear

2026-04-23T15:35:49Z

Obsidian: /* Source Locally */

Go back to [[emfcamp 2026 | EMFCamp 2026]] 
Take a look at what we brought to [[Weeze_Inventory|EMFCamp 2024]]

==Timeline==
Gear arrives with [[User:obsidian| Obsidian]] on Tuesday 14th July. ETA 15:00. 
Gear leaves with [[User:obsidian| Obsidian]] on Tuesday 21nd July ETD 13:00.

==Transport==
Obsidian transports all gear from DE and UK storage in a single [https://www.vandeburgwal.nl/huur-tandemas-gesloten-3-meter.-extra-breed-en-hoog trailer]. 
Dimensions of trailer are 306x154x180. 
This is probably oversized for our gear, but the length is necessary for dome struts and nothing else makes as much sense value wise from various dutch rental places.

==UK Storage==
*Dome
*Power
*???

==DE Storage==
*Rummary Tent
*Freeside Shelter
*[[Emfcamp 2026#Milliways_Requestable_Event_Shelters| Eventshelters]]
*Equipment for Dome
**Lights
***1 Clamp per light
***1 Safety wire loop per light
***1 Power cable per light
***1 DMX cable per light
**Speaker for chilldome (basic I/O nothing special)
**Camonets
***Tiewraps
**Bedsheets?
*Power
*Kitchen
**Oven
**Pots & Pans
**Cooking utensils?
**Induction Stove
**Cofeemaker
**???

==Source Locally==
*Sink Table for kitchen
** Depends on water situation, sent reminder to EMF on 22/04
*Stage for Chill Dome
** Requested info from local suppliers on 23/04
*Speaker for chilldome (basic I/O nothing special)
** Requested info from local suppliers on 23/04

[[Category:emfcamp2026]]

EMF2026Travelers

2026-04-21T07:28:30Z

Obsidian:

Go back to [[emfcamp 2026 | EMFCamp 2026]] 

EU creatures not yet used to post-brexit rules, remember to apply for an [https://www.gov.uk/eta ETA].

==Has a Ticket==

# boreq
# cqc
# obsidian
# OakKitten
# mc.fly
# cookingroffa
# Emerson
# Junglerot
# ChewyMoose
# dpk
# jobepunkt
# josie
# n0k0
# jelly
# foxboron
# zornem
# mara
# augeas
# meg
# Hackeriet/Norwegians
# Hackeriet/Norwegians +1
# Hackeriet/Norwegians +2
# Hackeriet/Norwegians +3
# Hackeriet/Norwegians +4
# Hackeriet/Norwegians +5
# Hackeriet/Norwegians +6
# Hackeriet/Norwegians +7
# Hackeriet/Norwegians +8
# wasamasa
# wasamasa +1
# lineargraph
# fellmoon
# fellmoon +1
# coderobe
# ax
# nyx
# Purple
# emy
# bloo
# leon
# merlos
# merlos +1

==Wants a Ticket==
# w1bble
# josie/cqc +1
# josie/cqc +1
# punycode (hase)

[[Category:emfcamp2026]]

EMF2026Gear

2026-04-21T07:26:43Z

Obsidian: /* DE Storage */

EMF2026Gear

2026-04-21T07:26:17Z

Obsidian:

Go back to [[emfcamp 2026 | EMFCamp 2026]] 
Take a look at what we brought to [[Weeze_Inventory|EMFCamp 2024]]

==Timeline==
Gear arrives with [[User:obsidian| Obsidian]] on Tuesday 14th July. ETA 15:00. 
Gear leaves with [[User:obsidian| Obsidian]] on Tuesday 21nd July ETD 13:00.

==Transport==
Obsidian transports all gear from DE and UK storage in a single [https://www.vandeburgwal.nl/huur-tandemas-gesloten-3-meter.-extra-breed-en-hoog trailer]. 
Dimensions of trailer are 306x154x180. 
This is probably oversized for our gear, but the length is necessary for dome struts and nothing else makes as much sense value wise from various dutch rental places.

==UK Storage==
*Dome
*Power
*???

==DE Storage==
*Rummary Tent
*Freeside Shelter
*[[Emfcamp 2026#Milliways_Requestable_Event_Shelters| Eventshelters]]
*Dome
**Lights
***1 Clamp per light
***1 Safety wire loop per light
***1 Power cable per light
***1 DMX cable per light
**Speaker for chilldome (basic I/O nothing special)
**Camonets
***Tiewraps
**Bedsheets?
*Power
*Kitchen
**Oven
**Pots & Pans
**Cooking utensils?
**Induction Stove
**Cofeemaker
**???

==Source Locally==
*Sink Table for kitchen
*Stage for Chill Dome

[[Category:emfcamp2026]]

Emfcamp 2026

2026-04-21T07:14:11Z

Obsidian: /* Meetings */

==General==
Electromagnetic Field 2026 will be held on 16–19 July 2026 at Eastnor Castle Deer Park, Eastnor.

Links:
* https://www.emfcamp.org/
* https://social.emfcamp.org/@emf

==Recent News==
* Next meeting: Sunday 26th April.

==Communication==
Matrix: join #milliways-emfcamp:milliways.info (and of course the milliways space: https://matrix.to/#/#milliways-space:milliways.info)

==Arrival and Departure==

The site is open from '''Thursday 16th July 10:00 to Monday 20th July 12:00 (midday)'''. You may not be on site outside of this time period unless you are helping with Milliways buildup/teardown and/or camp buildup/teardown.

If you want to be on site outside of that time period to help with Milliways buildup or teardown you need to be cleared by Obsidian who will explain how that works to you and tell you how early you can come or how late you can leave.

If you want to come even earlier or leave even later to help with camp buildup/teardown you need to be cleared by orga, consult [https://www.emfcamp.org/about/volunteering Volunteering] (you should sign up for the mailing list, when in doubt contact [mailto:volunteer@emfcamp.org]).

==Travelers==

'''Please add yourself''' so that we have a better idea of what the situation is: [[EMF2026Travelers | Travelers to emfcamp 2026]].

EU creatures not yet used to post-brexit rules, remember to apply for an [https://www.gov.uk/eta ETA].

==Timeline==

{| class="wikitable" style="margin:auto"
|+ Planned timeline subject to changes
|-
! Date !! 13:00 !! 20:00
|-
| 2026-07-14 (Day -1) || [[EMF2026Gear | Gear Arrives]] || Nothing
|-
| 2026-07-15 (Day 0) || Buildup || Buildup
|-
| 2026-07-16 (Day 1) || ? || ?
|-
| 2026-07-17 (Day 2) || ? || ?
|-
| 2026-07-18 (Day 3) || ? || ?
|-
| 2026-07-19 (Day 4) || ? || ?
|-
| 2026-07-20 (Day 5) || Teardown || Teardown
|-
| 2026-07-21 (Day 6) || [[EMF2026Gear | Gear Leaves]] || Nothing
|}

==Subassemblies==

{| class="wikitable" style="margin:auto"
! Name !! Description !! How to contact?
|-
| Freeside || Freeside is a loose conglomerate of hackers from around the globe. Cyberpunks, Stirner enthusiasts, beer drinkers, hedgehog admirers, and victims of the programming language Go. || boreq/cqc/willscott etc
|-
| Ministery of Chaos|| || yawnbox
|-
| Your Assembly here? || ??? || ???
|-
|}

==Milliways Requestable Event Shelters==

Aka tents which you can request to appear for you at the campsite. They will be taken from storage and brought just for you so you need to sign up before the event. Each tent fits up to two people, specify the number of people that will sleep in the particular tent so we know how many sleeping bags to grab. The tents come with everything you need e.g. a sleeping pad and sleeping bags.

{| class="wikitable" style="margin:auto"
|-
! Tent number !! Names of the people entitled to pick the tent up !! Number of sleeping bags !! Notes
|-
| 1 || coderobe || 2 ||
|-
|}

==Meetings==
Pads can be created under https://cpad.milliways.info/, select "code". Please link here as "view" when done and as "edit" in the relevant channels during the meeting. The meetings usually happen at '''2100 German time'''.

{| class="wikitable" style="margin:auto"
! When !! Where !! Description !! Notes
|-
| 2026-03-29 || [https://jitsi.milliways.info/milliways jitsi] || Kickoff Meeting || [https://cpad.milliways.info/code/#/2/code/view/zbUqSpcwUkWlG9t+E4yb6cOeGclQ7B1SKHVPoX+d+yM/ Meeting Minutes]
|-
| 2026-04-26 || [https://jitsi.milliways.info/milliways jitsi] || Monthly || [https://cpad.milliways.info/code/#/2/code/edit/vproN+FWPERkWIIr9y2SNlft/ Meeting Minutes]
|-
|}

==Plans==

To have fun.

[[EMF2026Gear | Gear needed]]

==Coin==

Doesn't exist yet.

[[Category:emfcamp2026]]

EMF2026Gear

2026-04-17T11:51:19Z

Obsidian: /* DE Storage */

Go back to [[emfcamp 2026 | EMFCamp 2026]] 
Take a look at what we brought to [[Weeze_Inventory|EMFCamp 2024]]

==Timeline==
Gear arrives with [[User:obsidian| Obsidian]] on Tuesday 14th July. ETA 15:00. 
Gear leaves with [[User:obsidian| Obsidian]] on Tuesday 21nd July ETD 13:00.

==Transport==
Obsidian transports all gear from DE and UK storage in a single [https://www.vandeburgwal.nl/huur-tandemas-gesloten-3-meter.-extra-breed-en-hoog trailer]. 
Dimensions of trailer are 306x154x180. 
This is probably oversized for our gear, but the length is necessary for dome struts and nothing else makes as much sense value wise from various dutch rental places.

==UK Storage==
*Dome
*Power
*???

==DE Storage==
*Rummary Tent
*Freeside Shelter
*[[Emfcamp 2026#Milliways_Requestable_Event_Shelters| Eventshelters]]
*Lights
**1 Clamp per light
**1 Safety wire loop per light
**1 Power cable per light
**1 DMX cable per light
*Speaker for chilldome (basic I/O nothing special)
*Camonets
**Tiewraps
*Bedsheets?
*Power
*Pots & Pans
*Cooking utensils?
*Induction Stove
*Cofeemaker
*???

[[Category:emfcamp2026]]

Emfcamp 2026

2026-03-29T20:02:29Z

Obsidian: /* Meetings */

==General==
Electromagnetic Field 2026 will be held on 16–19 July 2026 at Eastnor Castle Deer Park, Eastnor.

Links:
* https://www.emfcamp.org/
* https://social.emfcamp.org/@emf

Tickets:
* <s>Monday 9th March, 20:00 UTC</s>
* <s>Sunday 22nd March, 15:00 UTC</s>
* Thursday 2nd April, 18:00 BST

==Recent News==
* First meeting Sunday 29th.
* If you want a Milliways Tent add yourself in one of the sections below.

==Communication==
Matrix: join #milliways-emfcamp:milliways.info (and of course the milliways space: https://matrix.to/#/#milliways-space:milliways.info)

==Travelers==

'''Please add yourself''' so that we have a better idea of what the situation is: [[EMF2026Travelers | Travelers to emfcamp 2026]].

EU creatures not yet used to post-brexit rules, remember to apply for an [https://www.gov.uk/eta ETA].

==Timeline==

{| class="wikitable" style="margin:auto"
|+ Planned timeline subject to changes
|-
! Date !! 13:00 !! 20:00
|-
| 2026-07-14 (Day -1) || [[EMF2026Gear | Gear Arrives]] || Nothing
|-
| 2026-07-15 (Day 0) || Buildup || Buildup
|-
| 2026-07-16 (Day 1) || ? || ?
|-
| 2026-07-17 (Day 2) || ? || ?
|-
| 2026-07-18 (Day 3) || ? || ?
|-
| 2026-07-19 (Day 4) || ? || ?
|-
| 2026-07-20 (Day 5) || Teardown || Teardown
|-
| 2026-07-21 (Day 6) || [[EMF2026Gear | Gear Leaves]] || Nothing
|}

==Subassemblies==

{| class="wikitable" style="margin:auto"
! Name !! Description !! How to contact?
|-
| Freeside || Freeside is a loose conglomerate of hackers from around the globe. Cyberpunks, Stirner enthusiasts, beer drinkers, hedgehog admirers, and victims of the programming language Go. || boreq/cqc/willscott etc
|-
| Ministery of Chaos|| || yawnbox
|-
| Your Assembly here? || ??? || ???
|-
|}

==Milliways Requestable Event Shelters==

Aka tents which you can request to appear for you at the campsite. They will be taken from storage and brought just for you so you need to sign up before the event. Each tent fits up to two people, specify the number of people that will sleep in the particular tent so we know how many sleeping bags to grab. The tents come with everything you need e.g. a sleeping pad and sleeping bags.

{| class="wikitable" style="margin:auto"
|-
! Tent number !! Names of the people entitled to pick the tent up !! Number of sleeping bags !! Notes
|-
| 1 || coderobe || 2 ||
|-
|}

==Meetings==
Pads can be created under https://cpad.milliways.info/, select "code". Please link here as "view" when done and as "edit" in the relevant channels during the meeting. The meetings usually happen at '''2100 German time'''.

{| class="wikitable" style="margin:auto"
! When !! Where !! Description !! Notes
|-
| 2026-03-29 || [https://jitsi.milliways.info/milliways jitsi] || Kickoff Meeting || [https://cpad.milliways.info/code/#/2/code/view/zbUqSpcwUkWlG9t+E4yb6cOeGclQ7B1SKHVPoX+d+yM/ Meeting Minutes]
|-
| 2026-04-26 || || Monthly ||
|-
|}

==Plans==

To have fun.

[[EMF2026Gear | Gear needed]]

==Coin==

Doesn't exist yet.

[[Category:emfcamp2026]]

Emfcamp 2026

2026-03-26T13:45:10Z

Obsidian:

==General==
Electromagnetic Field 2026 will be held on 16–19 July 2026 at Eastnor Castle Deer Park, Eastnor.

Links:
* https://www.emfcamp.org/
* https://chaos.social/@emf@emfcamp.org

Tickets:
* <s>Monday 9th March, 20:00 UTC</s>
* <s>Sunday 22nd March, 15:00 UTC</s>
* Thursday 2nd April, 18:00 BST

==Recent News==
* First meeting Sunday 29th.
* If you want a Milliways Tent add yourself in one of the sections below.

==Communication==
Matrix: join #milliways-emfcamp:milliways.info (and of course the milliways space: https://matrix.to/#/#milliways-space:milliways.info)

==Travelers==

'''Please add yourself''' so that we have a better idea of what the situation is: [[EMF2026Travelers | Travelers to emfcamp 2026]].

EU creatures not yet used to post-brexit rules, remember to apply for an [https://www.gov.uk/eta ETA].

==Timeline==

{| class="wikitable" style="margin:auto"
|+ Planned timeline subject to changes
|-
! Date !! 13:00 !! 20:00
|-
| 2026-07-14 (Day -1) || [[EMF2026Gear | Gear Arrives]] || Nothing
|-
| 2026-07-15 (Day 0) || Buildup || Buildup
|-
| 2026-07-16 (Day 1) || ? || ?
|-
| 2026-07-17 (Day 2) || ? || ?
|-
| 2026-07-18 (Day 3) || ? || ?
|-
| 2026-07-19 (Day 4) || ? || ?
|-
| 2026-07-20 (Day 5) || Teardown || Teardown
|-
| 2026-07-21 (Day 6) || [[EMF2026Gear | Gear Leaves]] || Nothing
|}

==Subassemblies==

{| class="wikitable" style="margin:auto"
! Name !! Description !! How to contact?
|-
| Freeside || Freeside is a loose conglomerate of hackers from around the globe. Cyberpunks, Stirner enthusiasts, beer drinkers, hedgehog admirers, and victims of the programming language Go. || #freeside @ hackint
|-
| Your Assembly here? || ??? || ???
|-
|}

==Milliways Requestable Event Shelters==

Aka tents which you can request to appear for you at the campsite. They will be taken from storage and brought just for you so you need to sign up before the event. Each tent fits up to two people, specify the number of people that will sleep in the particular tent so we know how many sleeping bags to grab. The tents come with everything you need e.g. a sleeping pad and sleeping bags.

{| class="wikitable" style="margin:auto"
|-
! Tent number !! Names of the people entitled to pick the tent up !! Number of sleeping bags !! Notes
|-
| 1 || coderobe || 2 ||
|-
|}

==Meetings==
Pads can be created under https://cpad.milliways.info/, select "code". Please link here as "view" when done and as "edit" in the relevant channels during the meeting. The meetings usually happen at '''2100 German time'''.

{| class="wikitable" style="margin:auto"
! When !! Where !! Description !! Notes
|-
| 2026-03-29 || [https://jitsi.milliways.info/milliways jitsi] || Kickoff Meeting || [https://cpad.milliways.info/code/#/2/code/edit/dqAPcOpGPlisYNh7KlZjeqew/ Meeting Minutes]
|-
|}

==Plans==

To have fun.

[[EMF2026Gear | Gear needed]]

==Coin==

Doesn't exist yet.

[[Category:emfcamp2026]]

EMF2026Travelers

2026-03-26T13:44:55Z

Obsidian:

Go back to [[emfcamp 2026 | EMFCamp 2026]] 

EU creatures not yet used to post-brexit rules, remember to apply for an [https://www.gov.uk/eta ETA].

Add yourself if you have a ticket:

# boreq
# cqc
# obsidian
# OakKitten

[[Category:emfcamp2026]]

EMF2026Travelers

2026-03-26T10:39:53Z

Obsidian:

Go back to [[emfcamp 2026 | EMFCamp 2026]] 

Add yourself if you have a ticket:

# boreq
# cqc
# obsidian
# OakKitten

[[Category:emfcamp2026]]

Emfcamp 2026

2026-03-24T12:10:10Z

Obsidian:

==General==
Electromagnetic Field 2026 will be held on 16–19 July 2026 at Eastnor Castle Deer Park, Eastnor.

Links:
* https://www.emfcamp.org/
* https://chaos.social/@emf@emfcamp.org

Tickets:
* <s>Monday 9th March, 20:00 UTC</s>
* <s>Sunday 22nd March, 15:00 UTC</s>
* Thursday 2nd April, 18:00 BST

==Recent News==
* First meeting Sunday 29th.
* If you want a Milliways Tent add yourself in one of the sections below.

==Communication==
Matrix: join #milliways-emfcamp:milliways.info (and of course the milliways space: https://matrix.to/#/#milliways-space:milliways.info)

==Travelers==

[[EMF2026Travelers | Travelers to emfcamp 2026]]

==Timeline==

{| class="wikitable" style="margin:auto"
|+ Planned timeline subject to changes
|-
! Date !! 13:00 !! 20:00
|-
| 2026-07-14 (Day -1) || [[EMF2026Gear | Gear Arrives]] || Nothing
|-
| 2026-07-15 (Day 0) || Buildup || Buildup
|-
| 2026-07-16 (Day 1) || ? || ?
|-
| 2026-07-17 (Day 2) || ? || ?
|-
| 2026-07-18 (Day 3) || ? || ?
|-
| 2026-07-19 (Day 4) || ? || ?
|-
| 2026-07-20 (Day 5) || Teardown || Teardown
|-
| 2026-07-21 (Day 6) || [[EMF2026Gear | Gear Leaves]] || Nothing
|}

==Subassemblies==

{| class="wikitable" style="margin:auto"
! Name !! Description !! How to contact?
|-
| Freeside || Freeside is a loose conglomerate of hackers from around the globe. Cyberpunks, Stirner enthusiasts, beer drinkers, hedgehog admirers, and victims of the programming language Go. || #freeside @ hackint
|-
| Your Assembly here? || ??? || ???
|-
|}

==Milliways Requestable Event Shelters==

Aka tents which you can request to appear for you at the campsite. They will be taken from storage and brought just for you so you need to sign up before the event. Each tent fits up to two people, specify the number of people that will sleep in the particular tent so we know how many sleeping bags to grab. The tents come with everything you need e.g. a sleeping pad and sleeping bags.

{| class="wikitable" style="margin:auto"
|-
! Tent number !! Names of the people entitled to pick the tent up !! Number of sleeping bags !! Notes
|-
| 1 || coderobe || 2 ||
|-
|}

==Meetings==
Pads can be created under https://cpad.milliways.info/, select "code". Please link here as "view" when done and as "edit" in the relevant channels during the meeting. The meetings usually happen at '''2100 German time'''.

{| class="wikitable" style="margin:auto"
! When !! Where !! Description !! Notes
|-
| 2026-03-29 || [https://jitsi.milliways.info/milliways jitsi] || Kickoff Meeting || [https://cpad.milliways.info/code/#/2/code/edit/dqAPcOpGPlisYNh7KlZjeqew/ Meeting Minutes]
|-
|}

==Plans==

To have fun.

[[EMF2026Gear | Gear needed]]

==Coin==

Doesn't exist yet.

[[Category:emfcamp2026]]

Emfcamp 2026

2026-03-24T12:08:00Z

Obsidian: /* Meetings */

==General==
Electromagnetic Field 2026 will be held on 16–19 July 2026 at Eastnor Castle Deer Park, Eastnor.

Links:
* https://www.emfcamp.org/
* https://chaos.social/@emf@emfcamp.org

Tickets:
* <s>Monday 9th March, 20:00 UTC</s>
* <s>Sunday 22nd March, 15:00 UTC</s>
* Thursday 2nd April, 18:00 BST

==Recent News==
* First meeting Sunday 29th.
* If you want a Milliways Tent add yourself in one of the sections below.

==Communication==
Matrix: join #milliways-emfcamp:milliways.info (and of course the milliways space: https://matrix.to/#/#milliways-space:milliways.info)

==Travelers==

[[EMF2026Travelers | Travelers to emfcamp 2026]]

==Timeline==

{| class="wikitable" style="margin:auto"
|+ Planned timeline subject to changes
|-
! Date !! 13:00 !! 20:00
|-
| 2026-07-14 (Day -1) || [[EMF2026Gear | Gear Arrives]] || Nothing
|-
| 2026-07-15 (Day 0) || Buildup || Buildup
|-
| 2026-07-16 (Day 1) || ? || ?
|-
| 2026-07-17 (Day 2) || ? || ?
|-
| 2026-07-18 (Day 3) || ? || ?
|-
| 2026-07-19 (Day 4) || ? || ?
|-
| 2026-07-20 (Day 5) || Teardown || Teardown
|-
| 2026-07-21 (Day 6) || [[EMF2026Gear | Gear Leaves]] || Nothing
|}

==Subassemblies==

{| class="wikitable" style="margin:auto"
! Name !! Description !! How to contact?
|-
| Freeside || Freeside is a loose conglomerate of hackers from around the globe. Cyberpunks, Stirner enthusiasts, beer drinkers, hedgehog admirers, and victims of the programming language Go. || #freeside @ hackint
|-
| Your Assembly here? || ??? || ???
|-
|}

==Milliways Requestable Event Shelters==

Aka tents which you can request to appear for you at the campsite. They will be taken from storage and brought just for you so you need to sign up before the event. Each tent fits up to two people, specify the number of people that will sleep in the particular tent so we know how many sleeping bags to grab. The tents come with everything you need e.g. a sleeping pad and sleeping bags.

{| class="wikitable" style="margin:auto"
|-
! Tent number !! Names of the people entitled to pick the tent up !! Number of sleeping bags !! Notes
|-
| 1 || coderobe || 2 ||
|-
|}

==Meetings==
Pads can be created under https://cpad.milliways.info/, select "code". Please link here as "view" when done and as "edit" in the relevant channels during the meeting. The meetings usually happen at '''2100 German time'''.

{| class="wikitable" style="margin:auto"
! When !! Where !! Description !! Notes
|-
| 2026-03-29 || [https://jitsi.milliways.info/milliways jitsi] || Kickoff Meeting || [https://cpad.milliways.info/code/#/2/code/view/zbUqSpcwUkWlG9t+E4yb6cOeGclQ7B1SKHVPoX+d+yM/ Meeting Minutes]
|-
|}

==Plans==

To have fun.

[[EMF2026Gear | Gear needed]]

==Coin==

Doesn't exist yet.

[[Category:emfcamp2026]]

Emfcamp 2026

2026-03-24T09:46:50Z

Obsidian: /* Subassemblies */

==General==
Electromagnetic Field 2026 will be held on 16–19 July 2026 at Eastnor Castle Deer Park, Eastnor.

Links:
* https://www.emfcamp.org/
* https://chaos.social/@emf@emfcamp.org

Tickets:
* <s>Monday 9th March, 20:00 UTC</s>
* <s>Sunday 22nd March, 15:00 UTC</s>
* Thursday 2nd April, 18:00 BST

==Recent News==
* First meeting Sunday 29th.
* If you want a Milliways Tent add yourself in one of the sections below.

==Communication==
Matrix: join #milliways-emfcamp:milliways.info (and of course the milliways space: https://matrix.to/#/#milliways-space:milliways.info)

==Travelers==

[[EMF2026Travelers | Travelers to emfcamp 2026]]

==Timeline==

{| class="wikitable" style="margin:auto"
|+ Planned timeline subject to changes
|-
! Date !! 13:00 !! 20:00
|-
| 2026-07-14 (Day -1) || [[EMF2026Gear | Gear Arrives]] || Nothing
|-
| 2026-07-15 (Day 0) || Buildup || Buildup
|-
| 2026-07-16 (Day 1) || ? || ?
|-
| 2026-07-17 (Day 2) || ? || ?
|-
| 2026-07-18 (Day 3) || ? || ?
|-
| 2026-07-19 (Day 4) || ? || ?
|-
| 2026-07-20 (Day 5) || Teardown || Teardown
|-
| 2026-07-21 (Day 6) || [[EMF2026Gear | Gear Leaves]] || Nothing
|}

==Subassemblies==

{| class="wikitable" style="margin:auto"
! Name !! Description !! How to contact?
|-
| Freeside || Freeside is a loose conglomerate of hackers from around the globe. Cyberpunks, Stirner enthusiasts, beer drinkers, hedgehog admirers, and victims of the programming language Go. || #freeside @ hackint
|-
| Your Assembly here? || ??? || ???
|-
|}

==Milliways Requestable Event Shelters==

Aka tents which you can request to appear for you at the campsite. They will be taken from storage and brought just for you so you need to sign up before the event. Each tent fits up to two people, specify the number of people that will sleep in the particular tent so we know how many sleeping bags to grab. The tents come with everything you need e.g. a sleeping pad and sleeping bags.

{| class="wikitable" style="margin:auto"
|-
! Tent number !! Names of the people entitled to pick the tent up !! Number of sleeping bags !! Notes
|-
| 1 || coderobe || 2 ||
|-
|}

==Meetings==
Pads can be created under https://cpad.milliways.info/, select "code". Please link here as "view" when done and as "edit" in the relevant channels during the meeting. The meetings usually happen at '''2100 German time'''.

{| class="wikitable" style="margin:auto"
! When !! Where !! Description !! Notes
|-
| 2026-03-29 || [https://jitsi.milliways.info/milliways jitsi] || Kickoff Meeting || [https://example.com Meeting Minutes Todo]
|-
|}

==Plans==

To have fun.

[[EMF2026Gear | Gear needed]]

==Coin==

Doesn't exist yet.

[[Category:emfcamp2026]]

EMF2026Travelers

2026-03-24T09:45:37Z

Obsidian:

Go back to [[emfcamp 2026 | EMFCamp 2026]] 

Add yourself if you have a ticket:

# User1
# User2
# User3

EMF2026Travelers

2026-03-24T09:45:23Z

Obsidian:

Go back to [[emfcamp 2026 | EMFCamp 2026]] 
Travelers to emfcamp 2026

Add yourself if you have a ticket:

# User1
# User2
# User3

EMF2026Gear

2026-03-17T09:51:19Z

Obsidian:

Go back to [[emfcamp 2026 | EMFCamp 2026]] 
Take a look at what we brought to [[Weeze_Inventory|EMFCamp 2024]]

==Timeline==
Gear arrives with [[User:obsidian| Obsidian]] on Tuesday 14th July. ETA 15:00. 
Gear leaves with [[User:obsidian| Obsidian]] on Tuesday 21nd July ETD 13:00.

==Transport==
Obsidian transports all gear from DE and UK storage in a single [https://www.vandeburgwal.nl/huur-tandemas-gesloten-3-meter.-extra-breed-en-hoog trailer]. 
Dimensions of trailer are 306x154x180. 
This is probably oversized for our gear, but the length is necessary for dome struts and nothing else makes as much sense value wise from various dutch rental places.

==UK Storage==
*Dome
*Power
*???

==DE Storage==
*Rummary Tent
*Freeside Shelter
*[[Emfcamp 2026#Milliways_Requestable_Event_Shelters| Eventshelters]]
*Lights
**1 Clamp per light
**1 Safety wire loop per light
**1 Power cable per light
**1 DMX cable per light
*Power
*Pots & Pans
*Cooking utensils?
*Induction Stove
*Cofeemaker
*???

EMF2026Gear

2026-03-15T14:58:53Z

Obsidian: /* DE Storage */

Go back to [[emfcamp 2026 | EMFCamp 2026]]
Take a look at what we brought to [[Weeze_Inventory|EMFCamp 2024]]

==Timeline==
Gear arrives with [[User:obsidian| Obsidian]] on Tuesday 14th July. ETA 15:00. 
Gear leaves with [[User:obsidian| Obsidian]] on Tuesday 21nd July ETD 13:00.

==Transport==
Obsidian transports all gear from DE and UK storage in a single [https://www.vandeburgwal.nl/huur-tandemas-gesloten-3-meter.-extra-breed-en-hoog trailer]. 
Dimensions of trailer are 306x154x180. 
This is probably oversized for our gear, but the length is necessary for dome struts and nothing else makes as much sense value wise from various dutch rental places.

==UK Storage==
*Dome
*Power
*???

==DE Storage==
*Rummary Tent
*Freeside Shelter
*[[Emfcamp 2026#Milliways_Requestable_Event_Shelters| Eventshelters]]
*Lights
**1 Clamp per light
**1 Safety wire loop per light
**1 Power cable per light
**1 DMX cable per light
*Power
*Pots & Pans
*Cooking utensils?
*Induction Stove
*Cofeemaker
*???

EMF2026Gear

2026-03-15T14:57:20Z

Obsidian:

Go back to [[emfcamp 2026 | EMFCamp 2026]]
Take a look at what we brought to [[Weeze_Inventory|EMFCamp 2024]]

==Timeline==
Gear arrives with [[User:obsidian| Obsidian]] on Tuesday 14th July. ETA 15:00. 
Gear leaves with [[User:obsidian| Obsidian]] on Tuesday 21nd July ETD 13:00.

==Transport==
Obsidian transports all gear from DE and UK storage in a single [https://www.vandeburgwal.nl/huur-tandemas-gesloten-3-meter.-extra-breed-en-hoog trailer]. 
Dimensions of trailer are 306x154x180. 
This is probably oversized for our gear, but the length is necessary for dome struts and nothing else makes as much sense value wise from various dutch rental places.

==UK Storage==
*Dome
*Power
*???

==DE Storage==
*Rummary Tent
*Freeside Shelter
*[[Emfcamp 2026#Milliways_Requestable_Event_Shelters| Eventshelters]]
*Lights
*Power
*Pots & Pans
*Induction Stove
*Cofeemaker
*???

Emfcamp 2026

2026-03-15T14:54:45Z

Obsidian: /* Plans */

==General==
Electromagnetic Field 2026 will be held on 16–19 July 2026 at Eastnor Castle Deer Park, Eastnor.

Links:
* https://www.emfcamp.org/
* https://chaos.social/@emf@emfcamp.org

Tickets:
* <s>Monday 9th March, 20:00 UTC</s>
* Sunday 22nd March, 15:00 UTC
* Thursday 2nd April, 18:00 BST

==Recent News==
* There is no news.

==Communication==
Matrix: join #milliways-emfcamp:milliways.info (and of course the milliways space: https://matrix.to/#/#milliways-space:milliways.info)

==Timeline==

{| class="wikitable" style="margin:auto"
|+ Planned timeline subject to changes
|-
! Date !! 13:00 !! 20:00
|-
| 2026-07-14 (Day -1) || [[EMF2026Gear | Gear Arrives]] || Nothing
|-
| 2026-07-15 (Day 0) || Buildup || Buildup
|-
| 2026-07-16 (Day 1) || ? || ?
|-
| 2026-07-17 (Day 2) || ? || ?
|-
| 2026-07-18 (Day 3) || ? || ?
|-
| 2026-07-19 (Day 4) || ? || ?
|-
| 2026-07-20 (Day 5) || Teardown || Teardown
|-
| 2026-07-21 (Day 6) || [[EMF2026Gear | Gear Leaves]] || Nothing
|}

==Subassemblies==

{| class="wikitable" style="margin:auto"
! Name !! Description !! How to contact?
|-
| Freeside || || #freeside @ hackint
|-
|}

==Milliways Requestable Event Shelters==

Aka tents which you can request to appear for you at the campsite. They will be taken from storage and brought just for you so you need to sign up before the event. Each tent fits up to two people, specify the number of people that will sleep in the particular tent so we know how many sleeping bags to grab. The tents come with everything you need e.g. a sleeping pad and sleeping bags.

{| class="wikitable" style="margin:auto"
|-
! Tent number !! Names of the people entitled to pick the tent up !! Number of sleeping bags !! Notes
|-
| 1 || coderobe || 2 ||
|-
|}

==Meetings==
Pads can be created under https://cpad.milliways.info/, select "code". Please link here as "view" when done and as "edit" in the relevant channels during the meeting. The meetings usually happen at 2100 German time.

{| class="wikitable" style="margin:auto"
! When !! Where !! Description !! Notes
|-
| 2026-??-?? || [https://jitsi.milliways.info/milliways jitsi] || Orga Meeting || [https://example.com Meeting Minutes]
|-
|}

==Plans==

To have fun.

[[EMF2026Gear | Gear needed]]

==Coin==

Doesn't exist yet.

[[Category:emfcamp2026]]

EMF2026Gear

2026-03-15T14:53:57Z

Obsidian:

Go back to [[emfcamp 2026 | EMFCamp 2026]]

==Timeline==
Gear arrives with [[User:obsidian| Obsidian]] on Tuesday 14th July. ETA 15:00. 
Gear leaves with [[User:obsidian| Obsidian]] on Tuesday 21nd July ETD 13:00.

==Transport==
Obsidian transports all gear from DE and UK storage in a single [https://www.vandeburgwal.nl/huur-tandemas-gesloten-3-meter.-extra-breed-en-hoog trailer]. 
Dimensions of trailer are 306x154x180. 
This is probably oversized for our gear, but the length is necessary for dome struts and nothing else makes as much sense value wise from various dutch rental places.

==UK Storage==
*Dome
*Power
*???

==DE Storage==
*Rummary Tent
*Freeside Shelter
*Lights
*Power
*Pots & Pans
*Induction Stove
*Cofeemaker
*???
*[[Emfcamp 2026#Milliways_Requestable_Event_Shelters| Eventshelters]]

EMF2026Gear

2026-03-15T14:49:41Z

Obsidian:

Go back to [[emfcamp 2026 | EMFCamp 2026]]

==Timeline==
Gear arrives with [[User:obsidian| Obsidian]] on Tuesday 14th July
Gear leaves with [[User:obsidian| Obsidian]] on Tuesday 21nd July

==Transport==
Obsidian transports all gear from DE and UK storage in a single [https://www.vandeburgwal.nl/huur-tandemas-gesloten-3-meter.-extra-breed-en-hoog trailer]. 
Dimensions of trailer are 306x154x180. 
This is probably oversized for our gear, but the length is necessary for dome struts.

==UK Storage==
*Dome
*Power
*???

==DE Storage==
*Rummary Tent
*Freeside Shelter
*Lights
*Power
*Pots & Pans
*Induction Stove
*Cofeemaker
*???
*Eventshelters

EMF2026Gear

2026-03-15T14:42:32Z

Obsidian:

Go back to [[emfcamp 2026 | EMFCamp 2026]]

EMF2026Gear

2026-03-15T14:42:18Z

Obsidian: Created page with "Go back to EMFCamp 2026"

Go back to [[[[emfcamp 2026 | EMFCamp 2026]]

Emfcamp 2026

2026-03-15T14:41:14Z

Obsidian: /* Timeline */

==General==
Electromagnetic Field 2026 will be held on 16–19 July 2026 at Eastnor Castle Deer Park, Eastnor.

Links:
* https://www.emfcamp.org/
* https://chaos.social/@emf@emfcamp.org

Tickets:
* <s>Monday 9th March, 20:00 UTC</s>
* Sunday 22nd March, 15:00 UTC
* Thursday 2nd April, 18:00 BST

==Recent News==
* There is no news.

==Communication==
Matrix: join #milliways-emfcamp:milliways.info (and of course the milliways space: https://matrix.to/#/#milliways-space:milliways.info)

==Timeline==

{| class="wikitable" style="margin:auto"
|+ Planned timeline subject to changes
|-
! Date !! 13:00 !! 20:00
|-
| 2026-07-14 (Day -1) || [[EMF2026Gear | Gear Arrives]] || Nothing
|-
| 2026-07-15 (Day 0) || Buildup || Buildup
|-
| 2026-07-16 (Day 1) || ? || ?
|-
| 2026-07-17 (Day 2) || ? || ?
|-
| 2026-07-18 (Day 3) || ? || ?
|-
| 2026-07-19 (Day 4) || ? || ?
|-
| 2026-07-20 (Day 5) || Teardown || Teardown
|-
| 2026-07-21 (Day 6) || [[EMF2026Gear | Gear Leaves]] || Nothing
|}

==Subassemblies==

{| class="wikitable" style="margin:auto"
! Name !! Description !! How to contact?
|-
| Freeside || || #freeside @ hackint
|-
|}

==Milliways Requestable Event Shelters==

Aka tents which you can request to appear for you at the campsite. They will be taken from storage and brought just for you so you need to sign up before the event. Each tent fits up to two people, specify the number of people that will sleep in the particular tent so we know how many sleeping bags to grab. The tents come with everything you need e.g. a sleeping pad and sleeping bags.

{| class="wikitable" style="margin:auto"
|-
! Tent number !! Names of the people entitled to pick the tent up !! Number of sleeping bags !! Notes
|-
| 1 || coderobe || 2 ||
|-
|}

==Meetings==
Pads can be created under https://cpad.milliways.info/, select "code". Please link here as "view" when done and as "edit" in the relevant channels during the meeting. The meetings usually happen at 2100 German time.

{| class="wikitable" style="margin:auto"
! When !! Where !! Description !! Notes
|-
| 2026-??-?? || [https://jitsi.milliways.info/milliways jitsi] || Orga Meeting || [https://example.com Meeting Minutes]
|-
|}

==Plans==

To have fun.

==Coin==

Doesn't exist yet.

[[Category:emfcamp2026]]

Emfcamp 2026

2026-03-15T14:37:52Z

Obsidian: /* Timeline */

==General==
Electromagnetic Field 2026 will be held on 16–19 July 2026 at Eastnor Castle Deer Park, Eastnor.

Links:
* https://www.emfcamp.org/
* https://chaos.social/@emf@emfcamp.org

Tickets:
* <s>Monday 9th March, 20:00 UTC</s>
* Sunday 22nd March, 15:00 UTC
* Thursday 2nd April, 18:00 BST

==Recent News==
* There is no news.

==Communication==
Matrix: join #milliways-emfcamp:milliways.info (and of course the milliways space: https://matrix.to/#/#milliways-space:milliways.info)

==Timeline==

{| class="wikitable" style="margin:auto"
|+ Planned timeline subject to changes
|-
! Date !! 13:00 !! 20:00
|-
| 2026-07-14 (Day -1) || Gear Arrives || Nothing
|-
| 2026-07-15 (Day 0) || Buildup || Buildup
|-
| 2026-07-16 (Day 1) || ? || ?
|-
| 2026-07-17 (Day 2) || ? || ?
|-
| 2026-07-18 (Day 3) || ? || ?
|-
| 2026-07-19 (Day 4) || ? || ?
|-
| 2026-07-20 (Day 5) || Teardown || Teardown
|-
| 2026-07-21 (Day 6) || Gear Leaves || Nothing
|}

==Subassemblies==

{| class="wikitable" style="margin:auto"
! Name !! Description !! How to contact?
|-
| Freeside || || #freeside @ hackint
|-
|}

==Milliways Requestable Event Shelters==

Aka tents which you can request to appear for you at the campsite. They will be taken from storage and brought just for you so you need to sign up before the event. Each tent fits up to two people, specify the number of people that will sleep in the particular tent so we know how many sleeping bags to grab. The tents come with everything you need e.g. a sleeping pad and sleeping bags.

{| class="wikitable" style="margin:auto"
|-
! Tent number !! Names of the people entitled to pick the tent up !! Number of sleeping bags !! Notes
|-
| 1 || coderobe || 2 ||
|-
|}

==Meetings==
Pads can be created under https://cpad.milliways.info/, select "code". Please link here as "view" when done and as "edit" in the relevant channels during the meeting. The meetings usually happen at 2100 German time.

{| class="wikitable" style="margin:auto"
! When !! Where !! Description !! Notes
|-
| 2026-??-?? || [https://jitsi.milliways.info/milliways jitsi] || Orga Meeting || [https://example.com Meeting Minutes]
|-
|}

==Plans==

To have fun.

==Coin==

Doesn't exist yet.

[[Category:emfcamp2026]]

Emfcamp 2026

2026-03-15T14:35:53Z

Obsidian: /* Timeline */

==General==
Electromagnetic Field 2026 will be held on 16–19 July 2026 at Eastnor Castle Deer Park, Eastnor.

Links:
* https://www.emfcamp.org/
* https://chaos.social/@emf@emfcamp.org

Tickets:
* <s>Monday 9th March, 20:00 UTC</s>
* Sunday 22nd March, 15:00 UTC
* Thursday 2nd April, 18:00 BST

==Recent News==
* There is no news.

==Communication==
Matrix: join #milliways-emfcamp:milliways.info (and of course the milliways space: https://matrix.to/#/#milliways-space:milliways.info)

==Timeline==

{| class="wikitable" style="margin:auto"
|+ Planned timeline subject to changes
|-
! Date !! 13:00 !! 20:00
|-
| 2026-07-14 (Day -1) || Gear Arrives || Nothing
|-
| 2026-07-15 (Day 0) || Buildup || Buildup
|-
| 2026-07-16 (Day 1) || ? || ?
|-
| 2026-07-17 (Day 2) || ? || ?
|-
| 2026-07-18 (Day 3) || ? || ?
|-
| 2026-07-19 (Day 4) || ? || ?
|}

==Subassemblies==

{| class="wikitable" style="margin:auto"
! Name !! Description !! How to contact?
|-
| Freeside || || #freeside @ hackint
|-
|}

==Milliways Requestable Event Shelters==

Aka tents which you can request to appear for you at the campsite. They will be taken from storage and brought just for you so you need to sign up before the event. Each tent fits up to two people, specify the number of people that will sleep in the particular tent so we know how many sleeping bags to grab. The tents come with everything you need e.g. a sleeping pad and sleeping bags.

{| class="wikitable" style="margin:auto"
|-
! Tent number !! Names of the people entitled to pick the tent up !! Number of sleeping bags !! Notes
|-
| 1 || coderobe || 2 ||
|-
|}

==Meetings==
Pads can be created under https://cpad.milliways.info/, select "code". Please link here as "view" when done and as "edit" in the relevant channels during the meeting. The meetings usually happen at 2100 German time.

{| class="wikitable" style="margin:auto"
! When !! Where !! Description !! Notes
|-
| 2026-??-?? || [https://jitsi.milliways.info/milliways jitsi] || Orga Meeting || [https://example.com Meeting Minutes]
|-
|}

==Plans==

To have fun.

==Coin==

Doesn't exist yet.

[[Category:emfcamp2026]]

Emfcamp 2026

2026-03-15T14:35:36Z

Obsidian: /* Timeline */

==General==
Electromagnetic Field 2026 will be held on 16–19 July 2026 at Eastnor Castle Deer Park, Eastnor.

Links:
* https://www.emfcamp.org/
* https://chaos.social/@emf@emfcamp.org

Tickets:
* <s>Monday 9th March, 20:00 UTC</s>
* Sunday 22nd March, 15:00 UTC
* Thursday 2nd April, 18:00 BST

==Recent News==
* There is no news.

==Communication==
Matrix: join #milliways-emfcamp:milliways.info (and of course the milliways space: https://matrix.to/#/#milliways-space:milliways.info)

==Timeline==

{| class="wikitable" style="margin:auto"
|+ Planned timeline subject to changes
|-
! Date !! 13:00 !! 20:00
| 2026-07-14 (Day -1) || Gear Arrives || Nothing
|-
| 2026-07-15 (Day 0) || Buildup || Buildup
|-
| 2026-07-16 (Day 1) || ? || ?
|-
| 2026-07-17 (Day 2) || ? || ?
|-
| 2026-07-18 (Day 3) || ? || ?
|-
| 2026-07-19 (Day 4) || ? || ?
|}

==Subassemblies==

{| class="wikitable" style="margin:auto"
! Name !! Description !! How to contact?
|-
| Freeside || || #freeside @ hackint
|-
|}

==Milliways Requestable Event Shelters==

Aka tents which you can request to appear for you at the campsite. They will be taken from storage and brought just for you so you need to sign up before the event. Each tent fits up to two people, specify the number of people that will sleep in the particular tent so we know how many sleeping bags to grab. The tents come with everything you need e.g. a sleeping pad and sleeping bags.

{| class="wikitable" style="margin:auto"
|-
! Tent number !! Names of the people entitled to pick the tent up !! Number of sleeping bags !! Notes
|-
| 1 || coderobe || 2 ||
|-
|}

==Meetings==
Pads can be created under https://cpad.milliways.info/, select "code". Please link here as "view" when done and as "edit" in the relevant channels during the meeting. The meetings usually happen at 2100 German time.

{| class="wikitable" style="margin:auto"
! When !! Where !! Description !! Notes
|-
| 2026-??-?? || [https://jitsi.milliways.info/milliways jitsi] || Orga Meeting || [https://example.com Meeting Minutes]
|-
|}

==Plans==

To have fun.

==Coin==

Doesn't exist yet.

[[Category:emfcamp2026]]

Dome

2026-02-17T15:25:02Z

Obsidian: /* Bolts */

<gallery>
File:dome.jpg
File:dome_night.jpg
File:EMF24_Sunrise_Dome.jpg
File:Dome being built up at why 2025.jpg
</gallery>

There are two domes, one in the UK and one in the EU. Both domes are the same spec and have a radius of 4.75m. One is in [[Weeze_Storage]] and the second one is in UK Storage (unclear where - ask EMF people).

== Problems ==

=== EMF 2024 ===
A new tarp can be ruined after several strong gusts of wind.
We need ground ankers and 200m of high tensile rope for each dome to prevent this from happening again + improved plans on setting up the dome and adding a tarp so it's clear for newbies.

=== WHY 2025 ===

Multiple screws (4-5?) could not be removed during teardown and had to be cut with an angle grinder, it felt like it was more than normally.

The dome was anchored with relatively small rebar anchors in the fears that it will fly away. There is no possible scenario in which the dome will fly off in strong winds. If that were to happen we will likely be dead by then. To prevent this the anchors needed would be those 50-100cm massive pegs for big festival tents, not the tiny things that were used. Adding those tiny anchors just makes teardown more difficult and serves no practical purpose. I'm not running any simulations to prove that I'm right but I also suspect that there is no data to prove that anchors are needed.

The tarp was rubbing on a joint and developed a hole, this was fixed with duct tape.

== Buildup ==

=== Diagram ===
[[File:Dome diagram.gif]]

=== Struts ===

The dome is built out of struts.

{| class="wikitable" style="margin:auto"
|-
! Amount !! Color !! Length !! Notes
|-
| 35 || Blue || about 3 meters || "The long ones"
|-
| 30 || Red || about 2.66 meters || "The short ones"
|-
|}

There is also a box with bolts and required tools which is likely to be stored next to the struts as well as a tarp used to cover the dome.

=== Tools ===

17 wrench, 17 ratchet with extra long nut driver.

=== Bolts ===
Preference;
* [https://www.nen.nl/en/nen-en-15048-1-2016-en-222710 EN 15048] DIN 933 bolts in M10x150
* Matching nuts and washers

EN 15048 is for "Non preloaded structural bolting" 
DIN 933 determines the bolt has a full thread

These bolts are a good high strength steel meant for steel construction that doesn't move/shift/vibrate much. Basically tensile or shearing forces only.

Emergency use only:
* Materials:
** (ungraded or unknown) Zinc Galvanized Steel
** (ungraded or unknown) Stainless Steel

These materials can and will suffer from [https://en.wikipedia.org/wiki/Galling Galling] or [https://en.wikipedia.org/wiki/Cold_welding cold weld] when the dome is built. You will need an anglegrinder or saw to remove these bolts after use.

* Bolt types:
** DIN 931

This bolt type has a partial thread, meaning the dome struts cannot be properly fastened

== Tarp ==
@TODO

== Benches ==

=== Wood ===

[[:File:Milliways_Final_dome_benches.pdf | Measurements.pdf]]

=== Screws ===

Woodscrews 4.5mm diameter 4.5cm long.

like http://i.ebayimg.com/thumbs/images/m/mDJf7KpPBwFp1AFym1DO9kQ/s-l225.jpg

Dome

2026-02-17T15:21:38Z

Obsidian: /* Screws */

<gallery>
File:dome.jpg
File:dome_night.jpg
File:EMF24_Sunrise_Dome.jpg
File:Dome being built up at why 2025.jpg
</gallery>

There are two domes, one in the UK and one in the EU. Both domes are the same spec and have a radius of 4.75m. One is in [[Weeze_Storage]] and the second one is in UK Storage (unclear where - ask EMF people).

== Problems ==

=== EMF 2024 ===
A new tarp can be ruined after several strong gusts of wind.
We need ground ankers and 200m of high tensile rope for each dome to prevent this from happening again + improved plans on setting up the dome and adding a tarp so it's clear for newbies.

=== WHY 2025 ===

Multiple screws (4-5?) could not be removed during teardown and had to be cut with an angle grinder, it felt like it was more than normally.

The dome was anchored with relatively small rebar anchors in the fears that it will fly away. There is no possible scenario in which the dome will fly off in strong winds. If that were to happen we will likely be dead by then. To prevent this the anchors needed would be those 50-100cm massive pegs for big festival tents, not the tiny things that were used. Adding those tiny anchors just makes teardown more difficult and serves no practical purpose. I'm not running any simulations to prove that I'm right but I also suspect that there is no data to prove that anchors are needed.

The tarp was rubbing on a joint and developed a hole, this was fixed with duct tape.

== Buildup ==

=== Diagram ===
[[File:Dome diagram.gif]]

=== Struts ===

The dome is built out of struts.

{| class="wikitable" style="margin:auto"
|-
! Amount !! Color !! Length !! Notes
|-
| 35 || Blue || about 3 meters || "The long ones"
|-
| 30 || Red || about 2.66 meters || "The short ones"
|-
|}

There is also a box with bolts and required tools which is likely to be stored next to the struts as well as a tarp used to cover the dome.

=== Tools ===

17 wrench, 17 ratchet with extra long nut driver.

=== Bolts ===
Preference;
* [https://www.nen.nl/en/nen-en-14399-4-2015-en-204280 EN 14399] HV DIN 933 bolts in M10x150
* Matching nuts and washers

EN 14399 is for "Preloaded structural bolting" 
DIN 933 determines the bolt has a full thread

These bolts are a good high strength steel meant for steel construction that can undergo more than just tensile or shearing forces, like vibrations through use (think of bridges) or environment (wind).

* Note: There are two systems! HR (UK) and HV (Germany) - we try to use the HV System! It should be stamped on the bolt's head.

Good alternative;
* [https://www.nen.nl/en/nen-en-15048-1-2016-en-222710 EN 15048] DIN 933 bolts in M10x150
* Matching nuts and washers

EN 15048 is for "Non preloaded structural bolting"

These bolts are a good high strength steel meant for steel construction that doesn't move/shift/vibrate much. Basically tensile or shearing forces only.

Emergency use only:
* Materials:
** (ungraded or unknown) Zinc Galvanized Steel
** (ungraded or unknown) Stainless Steel

These materials can and will suffer from [https://en.wikipedia.org/wiki/Galling Galling] or [https://en.wikipedia.org/wiki/Cold_welding cold weld] when the dome is built. You will need an anglegrinder or saw to remove these bolts after use.

* Bolt types:
** DIN 931

This bolt type has a partial thread, meaning the dome struts cannot be properly fastened

== Tarp ==
@TODO

== Benches ==

=== Wood ===

[[:File:Milliways_Final_dome_benches.pdf | Measurements.pdf]]

=== Screws ===

Woodscrews 4.5mm diameter 4.5cm long.

like http://i.ebayimg.com/thumbs/images/m/mDJf7KpPBwFp1AFym1DO9kQ/s-l225.jpg

MilliwaysStack

2026-01-23T20:55:06Z

Obsidian: /* Shopping List */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
A more realistic MVP than the 1st version is:
* Networking
* Virtual machines
* Databases - mariaDB / PostgreSQL

=== e-MVP ===
What used to be in the first MVP, but now is actually recognized to be e-MVP;
* Kubernetes / docker
* object storage
* file systems
* Firewalling
* Someone something redis I guess
* container registry

=== super e-MVP ===
The super extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
** <s>iDRAC6 Enterprise card</s>
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]
** Extremely weird behavior, Dashboard will only load if Debug is set to True and compression is turned on.
** Completed 2025-01-21

==== Compute ====
* [https://docs.openstack.org/nova/2025.1/install/ Compute Service]
** [https://docs.openstack.org/nova/2025.1/install/compute-install-ubuntu.html Bad] Documentation
*** The guide entirely fails to mention that keeping default config in <code>[api_database]</code> will result in errors.
*** The guide entirely fails to mention that keeping default config in <code>[database]</code> will result in errors.
**** Yes, that's basically the same documentation error twice, but for two different options, on the same page.
* [https://docs.openstack.org/neutron/2025.1/install/ Networking Service]
* Extremely weird behaviour when linking up with control node. placement service decided password was wrong, it wasn't, nova scheduler and conducter wouldn't start, the "fix" is basically patience, no changes were made between it not working and it working and I have no idea why this now works.
* Completed 2025-01-21

==== Overall feelings ====
* Clunky and convoluted.
** The MVP environment with 1 control node and 1 compute node feels about as capable as a 4-bay enthusiast NAS running proxmox.
*** If I purely count only the time spent on control and compute nodes, I reckon it took me about 18hrs to do something I did in 20mins on my terramaster.
* Documentation is unacceptably bad.
** Not kidding, there is better documentation on running automated piracy software.
*** Heck, there's better documentation written by indian scam farms to trick your family members to run TeamViewer to run play store giftcard scams.
** There can and should no excuse at all for the level of sheer incompetence displayed in these docs.
** OpenStack's documentation is abysmal and the responsible parties deserve to be held accountable for this.
* Super bad initial impression, like, I would not, could not even, recommend this in any professional capacity.
* I am dearly hoping that with scalability, this thing outrips my terramaster fast, else I wonder if this is worth the time, effort, fuel, money and electricity I have pumped into it.
** I fully realize OpenStack and proxmox are not the same, Nova is basically comparable to proxmox and all other services or modules are extra functionality that proxmox does not aim to offer. Like, obviously, I am exaggerating out of frustration and emotion, but this MVP is basically an externally managed proxmox and taken at face value, I think I'd recommend proxmox over this. But! Let's see how this evolves and scales.
* Tempted to try SUSE' [https://harvesterhci.io/ Harvester HCI]
** Although perhaps this isn't comparable?
*** I'm getting mixed messages in both writeups and user experience. I'm worried this means the documentation is equally as bad and people just don't know what harvester actually is or can do.

== communications ==

MilliwaysStack

2026-01-22T18:44:25Z

Obsidian: /* e-MVP */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
A more realistic MVP than the 1st version is:
* Networking
* Virtual machines
* Databases - mariaDB / PostgreSQL

=== e-MVP ===
What used to be in the first MVP, but now is actually recognized to be e-MVP;
* Kubernetes / docker
* object storage
* file systems
* Firewalling
* Someone something redis I guess
* container registry

=== super e-MVP ===
The super extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]
** Extremely weird behavior, Dashboard will only load if Debug is set to True and compression is turned on.
** Completed 2025-01-21

==== Compute ====
* [https://docs.openstack.org/nova/2025.1/install/ Compute Service]
** [https://docs.openstack.org/nova/2025.1/install/compute-install-ubuntu.html Bad] Documentation
*** The guide entirely fails to mention that keeping default config in <code>[api_database]</code> will result in errors.
*** The guide entirely fails to mention that keeping default config in <code>[database]</code> will result in errors.
**** Yes, that's basically the same documentation error twice, but for two different options, on the same page.
* [https://docs.openstack.org/neutron/2025.1/install/ Networking Service]
* Extremely weird behaviour when linking up with control node. placement service decided password was wrong, it wasn't, nova scheduler and conducter wouldn't start, the "fix" is basically patience, no changes were made between it not working and it working and I have no idea why this now works.
* Completed 2025-01-21

==== Overall feelings ====
* Clunky and convoluted.
** The MVP environment with 1 control node and 1 compute node feels about as capable as a 4-bay enthusiast NAS running proxmox.
*** If I purely count only the time spent on control and compute nodes, I reckon it took me about 18hrs to do something I did in 20mins on my terramaster.
* Documentation is unacceptably bad.
** Not kidding, there is better documentation on running automated piracy software.
*** Heck, there's better documentation written by indian scam farms to trick your family members to run TeamViewer to run play store giftcard scams.
** There can and should no excuse at all for the level of sheer incompetence displayed in these docs.
** OpenStack's documentation is abysmal and the responsible parties deserve to be held accountable for this.
* Super bad initial impression, like, I would not, could not even, recommend this in any professional capacity.
* I am dearly hoping that with scalability, this thing outrips my terramaster fast, else I wonder if this is worth the time, effort, fuel, money and electricity I have pumped into it.
** I fully realize OpenStack and proxmox are not the same, Nova is basically comparable to proxmox and all other services or modules are extra functionality that proxmox does not aim to offer. Like, obviously, I am exaggerating out of frustration and emotion, but this MVP is basically an externally managed proxmox and taken at face value, I think I'd recommend proxmox over this. But! Let's see how this evolves and scales.
* Tempted to try SUSE' [https://harvesterhci.io/ Harvester HCI]
** Although perhaps this isn't comparable?
*** I'm getting mixed messages in both writeups and user experience. I'm worried this means the documentation is equally as bad and people just don't know what harvester actually is or can do.

== communications ==

MilliwaysStack

2026-01-22T18:43:57Z

Obsidian: /* MVP */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
A more realistic MVP than the 1st version is:
* Networking
* Virtual machines
* Databases - mariaDB / PostgreSQL

=== e-MVP ===
What used to be in the first MVP, but now is actually recognized to be e-MVP;
An actual extended MVP;
* Kubernetes / docker
* object storage
* file systems
* Firewalling
* Someone something redis I guess
* container registry

=== super e-MVP ===
The super extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]
** Extremely weird behavior, Dashboard will only load if Debug is set to True and compression is turned on.
** Completed 2025-01-21

==== Compute ====
* [https://docs.openstack.org/nova/2025.1/install/ Compute Service]
** [https://docs.openstack.org/nova/2025.1/install/compute-install-ubuntu.html Bad] Documentation
*** The guide entirely fails to mention that keeping default config in <code>[api_database]</code> will result in errors.
*** The guide entirely fails to mention that keeping default config in <code>[database]</code> will result in errors.
**** Yes, that's basically the same documentation error twice, but for two different options, on the same page.
* [https://docs.openstack.org/neutron/2025.1/install/ Networking Service]
* Extremely weird behaviour when linking up with control node. placement service decided password was wrong, it wasn't, nova scheduler and conducter wouldn't start, the "fix" is basically patience, no changes were made between it not working and it working and I have no idea why this now works.
* Completed 2025-01-21

==== Overall feelings ====
* Clunky and convoluted.
** The MVP environment with 1 control node and 1 compute node feels about as capable as a 4-bay enthusiast NAS running proxmox.
*** If I purely count only the time spent on control and compute nodes, I reckon it took me about 18hrs to do something I did in 20mins on my terramaster.
* Documentation is unacceptably bad.
** Not kidding, there is better documentation on running automated piracy software.
*** Heck, there's better documentation written by indian scam farms to trick your family members to run TeamViewer to run play store giftcard scams.
** There can and should no excuse at all for the level of sheer incompetence displayed in these docs.
** OpenStack's documentation is abysmal and the responsible parties deserve to be held accountable for this.
* Super bad initial impression, like, I would not, could not even, recommend this in any professional capacity.
* I am dearly hoping that with scalability, this thing outrips my terramaster fast, else I wonder if this is worth the time, effort, fuel, money and electricity I have pumped into it.
** I fully realize OpenStack and proxmox are not the same, Nova is basically comparable to proxmox and all other services or modules are extra functionality that proxmox does not aim to offer. Like, obviously, I am exaggerating out of frustration and emotion, but this MVP is basically an externally managed proxmox and taken at face value, I think I'd recommend proxmox over this. But! Let's see how this evolves and scales.
* Tempted to try SUSE' [https://harvesterhci.io/ Harvester HCI]
** Although perhaps this isn't comparable?
*** I'm getting mixed messages in both writeups and user experience. I'm worried this means the documentation is equally as bad and people just don't know what harvester actually is or can do.

== communications ==

MilliwaysStack

2026-01-22T18:42:04Z

Obsidian: /* e-MVP */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== super e-MVP ===
The super extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]
** Extremely weird behavior, Dashboard will only load if Debug is set to True and compression is turned on.
** Completed 2025-01-21

==== Compute ====
* [https://docs.openstack.org/nova/2025.1/install/ Compute Service]
** [https://docs.openstack.org/nova/2025.1/install/compute-install-ubuntu.html Bad] Documentation
*** The guide entirely fails to mention that keeping default config in <code>[api_database]</code> will result in errors.
*** The guide entirely fails to mention that keeping default config in <code>[database]</code> will result in errors.
**** Yes, that's basically the same documentation error twice, but for two different options, on the same page.
* [https://docs.openstack.org/neutron/2025.1/install/ Networking Service]
* Extremely weird behaviour when linking up with control node. placement service decided password was wrong, it wasn't, nova scheduler and conducter wouldn't start, the "fix" is basically patience, no changes were made between it not working and it working and I have no idea why this now works.
* Completed 2025-01-21

==== Overall feelings ====
* Clunky and convoluted.
** The MVP environment with 1 control node and 1 compute node feels about as capable as a 4-bay enthusiast NAS running proxmox.
*** If I purely count only the time spent on control and compute nodes, I reckon it took me about 18hrs to do something I did in 20mins on my terramaster.
* Documentation is unacceptably bad.
** Not kidding, there is better documentation on running automated piracy software.
*** Heck, there's better documentation written by indian scam farms to trick your family members to run TeamViewer to run play store giftcard scams.
** There can and should no excuse at all for the level of sheer incompetence displayed in these docs.
** OpenStack's documentation is abysmal and the responsible parties deserve to be held accountable for this.
* Super bad initial impression, like, I would not, could not even, recommend this in any professional capacity.
* I am dearly hoping that with scalability, this thing outrips my terramaster fast, else I wonder if this is worth the time, effort, fuel, money and electricity I have pumped into it.
** I fully realize OpenStack and proxmox are not the same, Nova is basically comparable to proxmox and all other services or modules are extra functionality that proxmox does not aim to offer. Like, obviously, I am exaggerating out of frustration and emotion, but this MVP is basically an externally managed proxmox and taken at face value, I think I'd recommend proxmox over this. But! Let's see how this evolves and scales.
* Tempted to try SUSE' [https://harvesterhci.io/ Harvester HCI]
** Although perhaps this isn't comparable?
*** I'm getting mixed messages in both writeups and user experience. I'm worried this means the documentation is equally as bad and people just don't know what harvester actually is or can do.

== communications ==

MilliwaysStack

2026-01-22T11:38:21Z

Obsidian: /* Overall feelings */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]
** Extremely weird behavior, Dashboard will only load if Debug is set to True and compression is turned on.
** Completed 2025-01-21

==== Compute ====
* [https://docs.openstack.org/nova/2025.1/install/ Compute Service]
** [https://docs.openstack.org/nova/2025.1/install/compute-install-ubuntu.html Bad] Documentation
*** The guide entirely fails to mention that keeping default config in <code>[api_database]</code> will result in errors.
*** The guide entirely fails to mention that keeping default config in <code>[database]</code> will result in errors.
**** Yes, that's basically the same documentation error twice, but for two different options, on the same page.
* [https://docs.openstack.org/neutron/2025.1/install/ Networking Service]
* Extremely weird behaviour when linking up with control node. placement service decided password was wrong, it wasn't, nova scheduler and conducter wouldn't start, the "fix" is basically patience, no changes were made between it not working and it working and I have no idea why this now works.
* Completed 2025-01-21

==== Overall feelings ====
* Clunky and convoluted.
** The MVP environment with 1 control node and 1 compute node feels about as capable as a 4-bay enthusiast NAS running proxmox.
*** If I purely count only the time spent on control and compute nodes, I reckon it took me about 18hrs to do something I did in 20mins on my terramaster.
* Documentation is unacceptably bad.
** Not kidding, there is better documentation on running automated piracy software.
*** Heck, there's better documentation written by indian scam farms to trick your family members to run TeamViewer to run play store giftcard scams.
** There can and should no excuse at all for the level of sheer incompetence displayed in these docs.
** OpenStack's documentation is abysmal and the responsible parties deserve to be held accountable for this.
* Super bad initial impression, like, I would not, could not even, recommend this in any professional capacity.
* I am dearly hoping that with scalability, this thing outrips my terramaster fast, else I wonder if this is worth the time, effort, fuel, money and electricity I have pumped into it.
** I fully realize OpenStack and proxmox are not the same, Nova is basically comparable to proxmox and all other services or modules are extra functionality that proxmox does not aim to offer. Like, obviously, I am exaggerating out of frustration and emotion, but this MVP is basically an externally managed proxmox and taken at face value, I think I'd recommend proxmox over this. But! Let's see how this evolves and scales.
* Tempted to try SUSE' [https://harvesterhci.io/ Harvester HCI]
** Although perhaps this isn't comparable?
*** I'm getting mixed messages in both writeups and user experience. I'm worried this means the documentation is equally as bad and people just don't know what harvester actually is or can do.

== communications ==

MilliwaysStack

2026-01-22T11:31:03Z

Obsidian: /* Overall feelings */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]
** Extremely weird behavior, Dashboard will only load if Debug is set to True and compression is turned on.
** Completed 2025-01-21

==== Compute ====
* [https://docs.openstack.org/nova/2025.1/install/ Compute Service]
** [https://docs.openstack.org/nova/2025.1/install/compute-install-ubuntu.html Bad] Documentation
*** The guide entirely fails to mention that keeping default config in <code>[api_database]</code> will result in errors.
*** The guide entirely fails to mention that keeping default config in <code>[database]</code> will result in errors.
**** Yes, that's basically the same documentation error twice, but for two different options, on the same page.
* [https://docs.openstack.org/neutron/2025.1/install/ Networking Service]
* Extremely weird behaviour when linking up with control node. placement service decided password was wrong, it wasn't, nova scheduler and conducter wouldn't start, the "fix" is basically patience, no changes were made between it not working and it working and I have no idea why this now works.
* Completed 2025-01-21

==== Overall feelings ====
* Clunky and convoluted.
** The MVP environment with 1 control node and 1 compute node feels about as capable as a 4-bay enthusiast NAS running proxmox.
*** If I purely count only the time spent on control and compute nodes, I reckon it took me about 18hrs to do something I did in 20mins on my terramaster.
* Documentation is unacceptably bad.
** Not kidding, there is better documentation on running automated piracy software.
*** Heck, there's better documentation written by indian scam farms to trick your family members to run TeamViewer to run play store giftcard scams.
** There can and should no excuse at all for the level of sheer incompetence displayed in these docs.
** OpenStack's documentation is abysmal and the responsible parties deserve to be held accountable for this.
* Super bad initial impression, like, I would not, could not even, recommend this in any professional capacity.
* I am dearly hoping that with scalability, this thing outrips my terramaster fast, else I wonder if this is worth the time, effort, fuel, money and electricity I have pumped into it.
** I fully realize OpenStack and proxmox are not the same, Nova is basically comparable to proxmox and all other services or modules are extra functionality that proxmox does not aim to offer. Like, obviously, I am exaggerating out of frustration and emotion, but this MVP is basically an externally managed proxmox and taken at face value, I think I'd recommend proxmox over this. But! Let's see how this evolves and scales.
* Tempted to try SUSE' [https://harvesterhci.io/ Harvester HCI]
** Although perhaps this isn't comparable

== communications ==

MilliwaysStack

2026-01-22T11:03:49Z

Obsidian: /* Overall feelings */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]
** Extremely weird behavior, Dashboard will only load if Debug is set to True and compression is turned on.
** Completed 2025-01-21

==== Compute ====
* [https://docs.openstack.org/nova/2025.1/install/ Compute Service]
** [https://docs.openstack.org/nova/2025.1/install/compute-install-ubuntu.html Bad] Documentation
*** The guide entirely fails to mention that keeping default config in <code>[api_database]</code> will result in errors.
*** The guide entirely fails to mention that keeping default config in <code>[database]</code> will result in errors.
**** Yes, that's basically the same documentation error twice, but for two different options, on the same page.
* [https://docs.openstack.org/neutron/2025.1/install/ Networking Service]
* Extremely weird behaviour when linking up with control node. placement service decided password was wrong, it wasn't, nova scheduler and conducter wouldn't start, the "fix" is basically patience, no changes were made between it not working and it working and I have no idea why this now works.
* Completed 2025-01-21

==== Overall feelings ====
* Clunky and convoluted.
** The MVP environment with 1 control node and 1 compute node feels about as capable as a 4-bay enthusiast NAS running proxmox.
*** If I purely count only the time spent on control and compute nodes, I reckon it took me about 18hrs to do something I did in 20mins on my terramaster.
* Documentation is unacceptably bad.
** Not kidding, there is better documentation on running automated piracy software.
*** Heck, there's better documentation written by indian scam farms to trick your family members to run TeamViewer to run play store giftcard scams.
** There can and should no excuse at all for the level of sheer incompetence displayed in these docs.
** OpenStack's documentation is abysmal and the responsible parties deserve to be held accountable for this.
* Super bad initial impression, like, I would not, could not even, recommend this in any professional capacity.
* I am dearly hoping that with scalability, this thing outrips my terramaster fast, else I wonder if this is worth the time, effort, fuel, money and electricity I have pumped into it.
* Tempted to try SUSE' [https://harvesterhci.io/ Harvester HCI]

== communications ==

MilliwaysStack

2026-01-21T21:52:25Z

Obsidian: /* Overall feelings */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]
** Extremely weird behavior, Dashboard will only load if Debug is set to True and compression is turned on.
** Completed 2025-01-21

==== Compute ====
* [https://docs.openstack.org/nova/2025.1/install/ Compute Service]
** [https://docs.openstack.org/nova/2025.1/install/compute-install-ubuntu.html Bad] Documentation
*** The guide entirely fails to mention that keeping default config in <code>[api_database]</code> will result in errors.
*** The guide entirely fails to mention that keeping default config in <code>[database]</code> will result in errors.
**** Yes, that's basically the same documentation error twice, but for two different options, on the same page.
* [https://docs.openstack.org/neutron/2025.1/install/ Networking Service]
* Extremely weird behaviour when linking up with control node. placement service decided password was wrong, it wasn't, nova scheduler and conducter wouldn't start, the "fix" is basically patience, no changes were made between it not working and it working and I have no idea why this now works.
* Completed 2025-01-21

==== Overall feelings ====
* Clunky and convoluted.
** The MVP environment with 1 control node and 1 compute node feels about as capable as a 4-bay enthusiast NAS running proxmox.
*** If I purely count only the time spent on control and compute nodes, I reckon it took me about 18hrs to do something I did in 20mins on my terramaster.
* Documentation is unacceptably bad.
** Not kidding, there is better documentation on running automated piracy software.
*** Heck, there's better documentation written by indian scam farms to trick your family members to run TeamViewer to run play store giftcard scams.
** There can and should no excuse at all for the level of sheer incompetence displayed in these docs.
** OpenStack's documentation is abysmal and the responsible parties deserve to be held accountable for this.
* Super bad initial impression, like, I would not, could not even, recommend this in any professional capacity.
* I am dearly hoping that with scalability, this thing outrips my terramaster fast, else I wonder if this is worth the time, effort, fuel, money and electricity I have pumped into it.

== communications ==

MilliwaysStack

2026-01-21T21:48:31Z

Obsidian: /* Compute */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]
** Extremely weird behavior, Dashboard will only load if Debug is set to True and compression is turned on.
** Completed 2025-01-21

==== Compute ====
* [https://docs.openstack.org/nova/2025.1/install/ Compute Service]
** [https://docs.openstack.org/nova/2025.1/install/compute-install-ubuntu.html Bad] Documentation
*** The guide entirely fails to mention that keeping default config in <code>[api_database]</code> will result in errors.
*** The guide entirely fails to mention that keeping default config in <code>[database]</code> will result in errors.
**** Yes, that's basically the same documentation error twice, but for two different options, on the same page.
* [https://docs.openstack.org/neutron/2025.1/install/ Networking Service]
* Extremely weird behaviour when linking up with control node. placement service decided password was wrong, it wasn't, nova scheduler and conducter wouldn't start, the "fix" is basically patience, no changes were made between it not working and it working and I have no idea why this now works.
* Completed 2025-01-21

==== Overall feelings ====
* Clunky and convoluted.
** The MVP environment with 1 control node and 1 compute node feels about as capable as a 4-bay enthusiast NAS running proxmox.
* Documentation is unacceptably bad.
** Not kidding, there is better documentation on running automated piracy software.
*** Heck, there's better documentation written by indian scam farms to trick your family members to run TeamViewer to run play store giftcard scams.
** There can and should no excuse at all for the level of sheer incompetence displayed in these docs.
** OpenStack's documentation is abysmal and the responsible parties deserve to be held accountable for this.
* Super bad initial impression, like, I would not, could not even, recommend this in any professional capacity.
* I am dearly hoping that with scalability, this thing outrips my terramaster fast, else I wonder if this is worth the time, effort, fuel, money and electricity I have pumped into it.

== communications ==

MilliwaysStack

2026-01-21T20:07:48Z

Obsidian: /* Compute */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]
** Extremely weird behavior, Dashboard will only load if Debug is set to True and compression is turned on.
** Completed 2025-01-21

==== Compute ====
* [https://docs.openstack.org/nova/2025.1/install/ Compute Service]
** [https://docs.openstack.org/nova/2025.1/install/compute-install-ubuntu.html Bad] Documentation
*** The guide entirely fails to mention that keeping default config in <code>[api_database]</code> will result in errors.
*** The guide entirely fails to mention that keeping default config in <code>[database]</code> will result in errors.
**** Yes, that's basically the same documentation error twice, but for two different options, on the same page.
* [https://docs.openstack.org/neutron/2025.1/install/ Networking Service]

== communications ==

MilliwaysStack

2026-01-21T17:24:52Z

Obsidian: /* Controller */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]
** Extremely weird behavior, Dashboard will only load if Debug is set to True and compression is turned on.
** Completed 2025-01-21

==== Compute ====
* [https://docs.openstack.org/nova/2025.1/install/ Compute Service]
**
* [https://docs.openstack.org/neutron/2025.1/install/ Networking Service]

== communications ==

MilliwaysStack

2026-01-21T16:10:12Z

Obsidian: /* Controller */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* various Networking agents
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]
** Extremely weird behavior, Dashboard will only load if Debug is set to True and compression is turned on.
** Completed 2025-01-21

== communications ==

MilliwaysStack

2026-01-21T15:12:46Z

Obsidian: /* Controller */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
** Completed 2025-01-21
* various Networking agents
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]

== communications ==

MilliwaysStack

2026-01-21T14:52:47Z

Obsidian: /* Controller */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
*** Configuring openvswitch_agent.ini is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
**** The guide attempts to make you configure the the name of the bridge connected to the underlying provider physical network but you have not yet created this bridge when the guide asks you for the name.
* various Networking agents
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]

== communications ==

MilliwaysStack

2026-01-21T14:47:28Z

Obsidian: /* Controller */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** More [https://docs.openstack.org/neutron/2025.1/admin/deploy-ovs-provider.html Bad] Documentation
*** The guide refers to configuring the Open vSwitch agent and offers more information which directly contradicts the guide.
**** The guide says to edit neutron.conf with <code>service_plugins = router</code>
**** The Open vSwitch agent example configuration for controllers says: "Disable service plug-ins because provider networks do not require any."
* various Networking agents
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]

== communications ==

MilliwaysStack

2026-01-21T11:16:01Z

Obsidian: /* Network */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation mandates it.

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
* various Networking agents
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]

== communications ==

MilliwaysStack

2026-01-21T11:15:26Z

Obsidian: /* Network */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5
** Vlan 15
*** [https://docs.openstack.org/neutron/2025.1/install/environment-networking-ubuntu.html Provider Network]
**** This is an OpenStack thing for the secondary Control and Compute node interfaces.
**** Currently [https://docs.openstack.org/neutron/2025.1/install/environment-networking-controller-ubuntu.html no IP] address assigned.
**** May change in future if documentation

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
* various Networking agents
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]

== communications ==

MilliwaysStack

2026-01-21T11:05:35Z

Obsidian: /* Shopping List */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** Control and Compute servers each have 3 open m.2 NVMe slots
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
* various Networking agents
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]

== communications ==

MilliwaysStack

2026-01-21T11:04:30Z

Obsidian: /* Asset List */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 2.5" 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 2.5" 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
* various Networking agents
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]

== communications ==

MilliwaysStack

2026-01-21T11:03:32Z

Obsidian: /* Shopping List */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add 2.5" SSDs
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
* various Networking agents
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]

== communications ==

MilliwaysStack

2026-01-21T11:00:26Z

Obsidian: /* Shopping List */

We want to run an OpenStack experiment

== The grander idea ==

We want to try out an installation of OpenStack to give people around milliways experience with running (on).

From an unnamed source we got 10 HPE servers. We will use 8 of them to run OpenStack on it. Storage is on a seperate machine.

=== MVP ===
The MVP would be:
* Kubernetes / docker
* object storage
* file systems
* Networking
* Virtual machines
* Firewalling
* Databases - mariaDB / PostgreSQL
* Someone something redis I guess
* container registry

=== e-MVP ===
The extended MVP would be:
* functional Monitoring & alerting
* autoscaling
* integration into milliways identity & access management authentik
* logging & alerting

== the software stack explained ==

OpenStack is a cloud framework stack that offeres AWS / Azure / GC alike services.

Most documentation is availible for Ubuntu & Red Hat. On the longer term an installation under NixOS might be feasable.

== Asset List ==
=== Rack ===
* 47U
* 950mm external depth
** 915mm internal depth
=== Consumables & Small Materials ===
* 1 x Samsung 860 EVO 2TB
* Assorted M2 - M3 screws
* Assorted mismatched bundle of M5 and M6 cagenuts and bolts
* SFPs
=== Switches ===
* 2 x Dell PowerConnect 7048R-RA
* 1 x Cisco 3560e
=== [[MilliwaysStack_Servers | Servers]] ===
* 1 Dell PowerEdge R710 server as storage
** 2 x X5570 2,93GHz
** 192GB RAM
** 6 x 3,5" bays
*** 6 x hotswap 3,5" drive sleds/brackets
** Drives
*** 1 x Samsung 850 EVO 500GB
**** for OS
**** Hidden in aftermarket [https://www.amazon.nl/dp/B083XJPCGL "Optical Drive"] adapter.
***We have more drives than bays, but not enough drives to make a nice or ideal configuration. As such, the Dell storage situation is likely temporary until we can figure out if we add more 12T or 10T or keep as-is.
**** 2 x Seagate Exos X18 12TB
**** 1 x Seagate Exos X18 10TB
**** 4 x WD Red 4TB
**** 4 x WD Green 3TB
** no rails
* 2 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** PCI Riser to 4* NVMe adapter
*** 1TB Crucial NVMe
** iLO4
*** It seems it accepts 35DPH-SVSXJ-HGBJN-C7N5R-2SS4W as activation key for iLO Advanced license?
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** Slide rails
* 8 x HPE proliant DL380 Gen 8
** 2 x E5-2620 v3 2,4GHz
** 384GB ram
** iLO4
** without hard drives but has 2,5" bays
*** no drive sleds/brackets available, only blanks
** 7 x slide rails

== Shopping List ==
It's ofc. sexy as all hell to buy memory, AI cards, flash storage and allsorts, but literally none of that will ever work if we don't have our Generic basics in order. While we prefer big donations go to big ticket items, many small ticket items unexpectedly add up in the long run. Please do not forget the generic basics!
* Generic Basics
** PDU
*** <s>Temporary 1U unmanaged PDU with 16A/230V C19 input and 1* C19 + 8* Type F outlet.</s>
*** Perfect: Managed Rack mountable PDU with CEE Red 16A/20A 400v input to C13/C14 + C19/C20 outlets.
**** [https://www.eaton.com/us/en-us/skuPage.PDU3XEVSR6G20.html Stupid expensive example]
*** Alternatively; a "normal" Serverrack PDU (still strong prefer managed) + 16A/20A 400v -> 16A 230V transform
** Network Cables
*** Some actual properly matching cables would be great
*** [Color]
**** [''Type''],[''Amount''],[''Length'']
** Power Cables
*** Some actual properly matching cables would be great
*** [''Type''],[''Amount''],[''Length'']
** Screws, Nuts, Bolts
*** <s>Assorted M2,M2.5,M3 Screws</s>
*** Some actual properly matching cage nuts\bolts would be great
** PCI Risers
*** <s>Single NVMe adapters</s>
*** Multi NVMe adapters
** KVM
*** PiKVM?
* Dell - Storage
** <s>2* Drive sleds</s>
** <s>New RAID Card that supports passthrough\JBOD</s>
** <s>2* SFF-8087 -> SFF-8087 Mini SAS Cable</s>
** Drives
*** <s>500GB SSD for OS</s>
*** <s>Bracket and SATA Cable Adapter for SSD</s>
*** <s>Technically not shopping, but for historical tracking;
**** Old Exos X16 2 x 12T and 1 x 10T were RMA'd and replaced with X18's</s>
*** 12T ?
* HP1 - Control
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* HP2 - Compute
** <s>1* PCI riser to 4*NVMe adapter</s>
** <s>1* 1TB NVMe</s>
* Flash Storage
** We'll need [https://www.amazon.de/-/en/dp/B07GCDH5D8 Drive Trays] for the HPs if we wanna add SSDs
** <s>1 x 2TB Samsung 860 EVO</s>

== Documentation ==
nb. this is quick 'n' dirty as I go along.
In the short-term future I'd much rather replace this adhoc documentation with something like NetBox.
=== Network ===
* Supernet 10.42.0.0/16
** Vlan 42
*** Interconnect
*** 10.42.0.0/30
**** Gateway 10.42.0.1
**** Milliways Core 10.42.0.2
** Vlan 5
*** Mgmt \ OOB
*** 10.42.1.0/24
**** Milliways Core 10.42.1.1
**** Dell iDRAC 10.42.1.5
**** Dell RAID Controller 10.42.1.6
**** HP 1 iLO 10.42.1.7
**** HP 2 iLO 10.42.1.8
** Vlan 10
*** Prod
*** 10.42.10.0/24
**** Milliways Core 10.42.10.1
**** Dell 10.42.10.2
**** HP 1 10.42.10.3
**** HP 2 10.42.10.5

=== Cable Mgmt ===
As there are some early ambitions to physically take this environment to events, perhaps we should seriously think about making our lives easier by already thinking about colorcoding connectivity. While this will help us connecting everything again at $event when we're sleepdeprived\drunk\explaining to newbies, this has the added effect of making it all look slightly more cooler than just a spaghetti of all boring white cables or worse, a spaghetti of whatever the fuck we have lying around.

This is all just made-up without too much thought. This is specifically intended to start a discussion so we can work toward an agreement, it is not intended to be a unilateral decision. Example; You'll notice 0 thought was put into fiber or not ;)
* RED
** Mgmt \ OOB
*** iDRACs, iLOs, RAID Cards, etc
* GREEN
** Storage Prod
*** At least the Dell, maybe HPs if we get into flash storage
* BLUE
** Compute Prod
*** Likely overwhelmingly the HPs
* YELLOW
** Interconnect
*** Connectivity to $outside, between switches, whatever

=== Naming Convention ===
We need names!
Can't keep calling these "Dell", "HP1", "HP2" etc.
Calling them by their S/Ns is also super boring and cumbersome; "Oh yea, we need to setup 5V6S064"
We could even opt for dual names. Internally, when logged in to $shell, the names could be functional "milliways-control-node-1" so it's clear what you're doing, but externally, the Asset Tag could be a Hitchhiker's Guide to the Galaxy character or a DiscWorld town or something. That way, if we do ever show this off at events, we can do cool shit with light up tags, make stuff funny and recognizable and cool to talk about - it also makes it way more relatable to market for when asking for donations; "Ya, we're looking for extra storage for Überwald" sounds much better than "Ya we're looking for extra storage for 5V6S064 or milliways-control-node-1"
Naturally, once we get NetBox going, we can map the Asset names to the actual server name and potentially it's serial so we don't get confused internally (if we want to use serials, there's somethign to be said for not using serials here)

* Functional
** milliways-control-node-1
** milliways-control-node-2
** control-node-1
** compute-node-1
** flash-storage-1

* Marketing
** HGttG characters
*** Arthur
*** Ford
*** Zaphod
** Discworld locations
*** Ankh-Morpork
*** Überwald
*** Lancre
***

=== OpenStack ===

We're using 2025.1 (epoxy) as 2025.2 (flamingo) has an undocumented breaking change making installation of keystone impossible. We have registered a bug with the documentation on launchpad for this.

* [https://docs.openstack.org/install-guide/ Installation guide]
* [https://docs.openstack.org/install-guide/openstack-services.html#minimal-deployment-for-2025-1-epoxy Minimal Deployment]
* [https://docs.openstack.org/install-guide/overview.html#example-architecture Example Architecture]
Following installation guide recommendation, passwords are created with <code>openssl rand -hex 10</code> and saved in a password store.

==== Controller ====
* [https://docs.openstack.org/keystone/2025.1/install/ Identity service]
** [https://docs.openstack.org/keystone/2025.2/install/keystone-users-ubuntu.html Broken] in 2025.2
***[https://opendev.org/openstack/keystone/src/commit/82c80dccf6c2e74e27b90f5204de6da1fc6bd76d/releasenotes/notes/remove-wsgi-scripts-615b97ee4d6e0de2.yaml This] commit removes the WSGI scripts, ``keystone-wsgi-admin`` and ``keystone-wsgi-public``.
*** Both scripts are still called by the openstack command. This means running any openstack command to create a domain, projects, users, and roles fails with the error
****<code>Failed to discover available identity versions when contacting http://controller:5000/v3. Attempting to parse version from URL.</code>
*** Evidence:
**** <code>tail /var/log/apache2/keystone.log</code>
***** <code>Target WSGI script not found or unable to stat: /usr/bin/keystone-wsgi-public</code>
** Workaround, use 2025.1 instead
** Completed 2025-01-18
* [https://docs.openstack.org/glance/2025.1/install/ Image service]
** [https://docs.openstack.org/glance/2025.1/install/install-ubuntu.html Bad] Documentation
*** The guide has you create 3 API endpoints for the service.
**** You need to configure access to keystone with one of them, but you are not told which one. Only <code>public</code> will work.
*** Configuring glance-api.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
** Completed 2025-01-19
* [https://docs.openstack.org/placement/2025.1/install/ Placement service]
** [https://docs.openstack.org/placement/2025.1/install/verify.html Bad] Documentation
*** If you followed the guide, your user account [https://storyboard.openstack.org/#!/story/2008969 does not have the rights] to read <code>/etc/placement/placement.conf</code>
*** Running <code>placement-status upgrade check</code> as root proves the service works.
*** Undocumented requirement fulfilled; <code>usermod -aG placement</code>
** Completed 2025-01-20
* management portions of [https://docs.openstack.org/nova/2025.1/install/ Compute]
** [https://docs.openstack.org/nova/2025.1/install/controller-install-ubuntu.html Bad] Documentation
*** Configuring nova.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
*** The guide attempts to make you configure options for the networking service, which you have not installed yet, because the guide makes you install this service first
*** <code>Due to a packaging bug, remove the log_dir option from the [DEFAULT] section.</code>
**** ???? THEN FIX THE PACKAGE?!?!?!!!!
*** The <code>[glance]</code> option you are instructed to use is deprecated
** Completed 2025-01-20
* management portion of [https://docs.openstack.org/neutron/2025.1/install/ Networking]
** [https://docs.openstack.org/neutron/2025.1/install/controller-install-option2-ubuntu.html Bad] Documentation
*** Configuring neutron.conf is done haphazardly in the guide
**** config options are organized alphabetically, the guide is not.
* various Networking agents
* [https://docs.openstack.org/horizon/2025.1/install/ Dashboard]

== communications ==