Re: booting from a raid1

[lee]

On Tue, Oct 05, 2010 at 11:22:35PM -0400, Tom H wrote:
> On Mon, Oct 4, 2010 at 9:46 AM, lee <address@hidden> wrote:
> > On Sun, Oct 03, 2010 at 04:23:55PM -0400, Tom H wrote:
> >>
> >> 1. I forgot to ask earlier, are you running "grub-install /dev/sdd"
> >> (and/or sdf)?
> >
> > Yes, both --- it creates a /boot directory when I'm using another
> > partition for /boot, but no grub.cfg.
> 
> 
> >> 2. Differences with setups that I've used:
> >>
> >> 2.a. I've never used the mdadm mdp option (I use md) or the fdisk da
> >> option (I use fd).
> >
> > Hm, I don't know what these options are.
> 
> When I create an array, I create 6 raid autodetect (fd) partitions,
> sda1, sda2, sda3, sdb1, sdb2, sdb3 and then either create a1/b1,
> a2/b2, a3/b3 sets for "/", "/boot", swap with "mdadm --create ...
> --auto=md ..." or do so through d-i.

Yes, that's one way to do it: First partition physical disks and then
create md-devices from the partitions. That leaves you with a lot of
md-devices in addition to the disks: Not what I wanted :)

The other way is to create an md-device from physical disks and then
partition the md-device. That leaves you with one md device in
addition to your disks.

There's a variant by first creating partitions across the whole
physical devices, then creating an md-device from those partitions and
then partitioning the md-device. I've done that because it allows you
to specify where the actual data on the physical disks starts: That
can be important if you want to install grub on the disks which
eventually tells you that there isn't enough room available when the
first partition starts, for example, on sector 4. I've had that
problem with the 500GB disks.

Recent versions of fdisk seem to default to starting the first
partition at sector 2048. This might have to do with disks using 4k
sectors. The 2TB disks I got have 4k sector size, so I partitioned
them to avoid problems with sector alignment before creating the
RAID5. And since no partitions are needed otherwise, I didn't
partition the md-device but just created a filesystem on it.

> I tried to re-create your setup in VBox. I looked for an option to
> create a partitioned md device through d-i but it can only create a
> non-partitioned device AFAICT. I also created a partitioned md device
> using another install but I couldn't figure out, after running the
> initial mdadm creation command, what to do next to carve out the
> partitions and size them before booting from d-i to perform
> installation. I must be missing something crucial here but no amount
> of reading mdadm's man page or googling has helped me figure out the
> next step... And d-i only sees one raid device.

The Debian way seems to be to use LVM ... I think the d-i can create a
partitioned RAID, but I don't remember exactly. But it will fail to
install grub when you do that, I tried ...

Anyway, once you create an md-device, like /dev/md0, just use fdisk on
it (as in "fdisk /dev/md0"). You can partition it just like a physical
block device.

You've gone to great lengths trying to help --- thank you very much!

> As a WAG, I'd suggest that partitioned mdraid devices aren't supported
> by grub2 (and maybe even grub1). But it's just a total WAG. I'm sure
> that someone on the mdraid list would know.

Good idea to ask there ...

> >> 2.b. You're booted from sde1 and mount the md0p2 install's /tmp, /usr,
> >> /var, /home, and /opt over the sde1 install's equivalents before
> >> bind-mounting to the "/mnt/raid" mount of md0p2, and chrooting to it.
> >> I would've mounted them directly to "/mnt/raid".
> >
> > sde1 is only the root partition, /tmp, /usr, /var and /opt are on
> > partitions of md0, and /home is on md126. md126 is not partitioned. On
> > /mnt/raid, the partition that needs to become the new root partition
> > is mounted. Then the other partitions are bind-mounted to respective
> > directories under /mnt/raid so that I can chroot to
> > /mnt/raid. Mounting them directly to /mnt/raid would require to mount
> > them twice.
> 
> I understand that /tmp, /usr, /var, and /opt are on md partitions.
> What I don't understand, is that they are both the /tmp, /usr, /var,
> and /opt of "/" on sde1 and of "/" on md0p2 and furthermore, in the
> case of sde1, "/" is a single disk partition and /tmp, /usr, /var, and
> /opt are mdraid devices (strange - to me - in and of itself).

That's what the bind-mount is for: I was reading some wiki page about
using chroot, and it said you can/should use the bind option of mount
to make filesystems available inside the chroot environment as
needed.

I could probably boot in maintenance mode and then mount the other
partitions to their respective mount points under /mnt/raid, but you
can't just unmount them on a running system.

I was merely trying to trick grub into thinking that md0p2, the new
root partition, already is the new root partition by using chroot.

> If you want to have "/boot" on a separate non-mdadm'd disk, say sdg
> (I've lost track of your disks but I think that it is the next
> letter...), "/boot" can be "/dev/sdg1" and you'd then run
> "grub-install /dev/sdg".

Meanwhile, I got the SATA cables I needed. So now I have an SATA disk
connected I can try to use to put a /boot partition on. But when
grub-install doesn't create a grub.cfg --- which it doesn't --- it
still won't work. And there could still be a problem with getting
/dev/md0 up and running so that /dev/md0p2 can be used as the root
partition.

Hmm ... Ok, I just tried with a separate /boot partition mounted
directly under /boot on the root partition, but grub-install doesn't
create a grub.cfg.

The only way is probably to move the whole root partition to that SATA
disk and boot from that. At least that will allow me to remove the IDE
disk, but I won't have the root partition on a RAID.

So much to RAID support in grub :(( I wonder if I should file a bug
report? It's a bug that should be fixed before the next release of
Debian stable since it prevents installation altogether.