Archive for Linux

Vmware server + Fedora Core 6

VMWware server 1.0.1 build-29996 can not install flawlessly on FC6 with kernel 2.6.19-1.2911.fc6. Need some simple hack.

* Linux kernel version magic dismatch. You will dmesg show your something like “vmmon: version magic ‘2.6.19-1.2911.fc6 SMP mod_unload 686 REGPARM 4KSTACKS ‘ should be ‘2.6.19-1.2911.fc6 SMP mod_unload 586 REGPARM 4KSTACKS “. I modified the include/asm/module.h file, changed #elif defined CONFIG_M686 #define MODULE_PROC_FAMILY “686 ” to “586 “

* module compilation failure. change vmnet source by following also need to link autoconf.h to config.h under include/linux in kernel source code.

That’s it!

Leave a Comment

Linux kernel, kgdb, vmware

Setup kgdb with supported kernel is quite straight forward by following the document. But there do have some small roadblocks, especially when you try to run the Linux as a VM in VMWare server.

  • With 2.6.8 or later kernel, you will not see “Waiting for connection from remote gdb…”, as pointed out at here. I was panic for a while when I could not figure out where the kernel stopped at;
  • Setup VMWare virtual serial port by following this.
  • Tricky: by default, kgdb over serial will use serial port 1 in CONFIG_KGDB_PORT_NUM, which is 2nd serial port. Had better change this to serial port 0, which is 1st serial port. The reason is we, at least I, usually add only 1 serial port in VMWare server…


  • kdgb cvs 02/07/2007 version can apply cleanly on 2.6.17 and runs ok.
  • kgdb latest can be found via “git clone git://”, as of 02/07/07, it is based on 2.6.18

Leave a Comment

Linux serial console with VMWare server.

Linux has a complete remote serial console how to available. So why need another one?

  • I just need a really simple guide to finish my work. And later when my 256KB main brain memory overflow, I can pick it up quickly.
  • I also want to run Linux virtual machine within VMWare server using virtual serial console support from VMWare.

Let us call the Linux virtual machine as “target” and Linux host running VMWare as “host”:

Target side: [most c&p from previously mentioned howto]

  1. To enable kernel support. Select Y for -> Device Drivers-> Character devices-> Serial drivers-> 8250/16550 and compatible serial support (SERIAL_8250 [=y]). Recompile kernel.
  2. To add kernel run time parameter. Modify grub.conf and use parameter like “title root (hd0,0) kernel /vmlinuz- ro root=/dev/sda3 console=ttyS0,9600n8 console=tty0″. These are recommended parameters. 9600 baud rate, no parity bit, 8 bit data bit, 1 stop bit, no flow control. Also leave the original tty0 usable. Of course, faster baud rate like 57600 will be better.
  3. Check and make sure mgetty is installed.
  4. To enable login prompt. Add one line in /etc/inittab “S0:23:respawn:/sbin/mgetty ttyS0″. Modify /etc/mgetty+sendfax/mgetty.config. Add “port ttyS0 speed 9600 direct yes data-only yes toggle-dtr yes need-dsr yes port-owner root port-group root port-mode 600 login-prompt @ \P login:40 login-time 60 term vt102″
  5. To enable root login. Add ttyS0 to /etc/securetty.

VMWare VM setting:

  1. [C&P from here]. Power down your virtual machine and click “Edit Virtual Machine Settings”. Add a serial port, select “Connect to named pipe”, and provide the pipe name “/tmp/com_1″. (note that this isn’t really a named pipe, but a Unix-domain socket) Click “advanced settings” and set “yield CPU on poll”. You can also get to this setting from the device list once you finish adding the port.
  2. You can start the Linux VM now.

Host side:

  1. Install socat from here.
  2. run “socat -d -d /tmp/com_1 PTY:”. socat should report something like “PTY is /dev/pts/x”.
  3. start minicom, configure it with right serial port setting, save the configuration and run it.
  4. now from minicom should get a full functional console. Sometime need to be patient to wait the login console appear.

Note: VMWare allows a serial port to be (1) physical port, (2) output to file, and (3) named pipe. So if we only need to capture console output like oops or boot message, we can use (2) to output to foo, and do “tail -f foo” in host.

Comments (5)

Another IET recipe

Steffen shared another IET recipe with some nice features:

  • active/passive storage controller
  • configuration management across cluster
  • dynamic updating of running ietd environment
  • dynamically add new targets and LUNs
  • handle size changes of LUNs
  • logical volume backup

Leave a Comment

Zero-copy user space page access from Linux kernel

Recently we discussed about this SCST user space device handler, so fast accessing user space page become a key factor to success. Lucky we have this already.

Leave a Comment


A nice writeup on DRBD list about how to setup a HA configuration for Xen.

Leave a Comment

Virtual ROM support

Recently I added a virtual ROM support to IET, details can be found here. One simple use is to export various OS installation ISO to ESX or Xen, so you can install OS easier.

Next step is to allow ISO files dynamic “ejected” and “loaded”, and a virtual DVD writer support. The code is done, just need to be polished. A fun but illegal use of this code is to clone a DVD .. to system as a ISO file, then you can watch it later at any time. Of course, when you have XVID and VIDX, you don’t need that right? ;)

Seriously, this can be used as WORM archiving purpose. Or do a disk staging for D2D2H.

Leave a Comment

Another happy IET user

See how much people can save with OSS like IET.

” No more W2K-based “LanShare” servers for 50K Euros that crash every 2nd day. Just a Webinterface and a choice of emulated parallel SCSI (on FreeBSD) or iSCSI (Linux/IET).

The typical replacement for a REALLY crappy 50K Euro Lanshare has a hardware fingerprint of some 5K (using really good Hardware) … and you don’t need windows ;-)

So what you are waiting for? ;)

Leave a Comment

Linux kernel module reference count.

Need to check Linux kernel module reference count. So add a dump_stack to places (here and here) when get a reference on module.

If a Linux kernel module export a character device (similarly for block device), and a user space application open the device, how many reference count will add? I thought it will be one, but answer is two.

[<c0103350>] show_trace+0xd/0xf
[<c0103422>] dump_stack+0x17/0x19
[<c0155d16>] cdev_get+0x13/0x4d
[<c0155f55>] exact_lock+0xa/0x11
[<c023c7cd>] kobj_lookup+0xb3/0x133
[<c0155dc6>] chrdev_open+0x49/0x142
[<c014daaa>] __dentry_open+0xbc/0x182
[<c014dc39>] nameidata_to_filp+0x1c/0x2e
[<c014db9e>] do_filp_open+0x2e/0x35
[<c014de1e>] do_sys_open+0x38/0x6d
[<c014de69>] sys_open+0x16/0x18
[<c0102579>] sysenter_past_esp+0x56/0x8d

[<c0103350>] show_trace+0xd/0xf
[<c0103422>] dump_stack+0x17/0x19
[<c0155e6c>] chrdev_open+0xef/0x142
[<c014daaa>] __dentry_open+0xbc/0x182
[<c014dc39>] nameidata_to_filp+0x1c/0x2e
[<c014db9e>] do_filp_open+0x2e/0x35
[<c014de1e>] do_sys_open+0x38/0x6d
[<c014de69>] sys_open+0x16/0x18
[<c0102579>] sysenter_past_esp+0x56/0x8d


Leave a Comment


blktrace is a tool developed by Jens Axboe to trace the Linux io request. Here is a nice blktrace guide about installation and use. I have a live example to show why blktrace is useful.

Andre post a new io mode, blockio, for IET last month. The code runs well for him with various Linux iSCSI initiator. Later Ross found one problem with it. A Windows host can not create file system on any iSCSI volume export with blockio mode while Linux host can. And more funny thing is once a NTFS or FAT32 partition is created in advanced on an iSCSI volume, the Windows host can recognize the file system and do various file system operation without problem.

So what is the problem?

We used Wireshark to get all the iSCSI traffic. Windows format will read immediately after a write and the data read out is not the same as previous write. This confused the Window and lead to failure. But why normal I/O after format can read correct data out? We need a way to find out what happened exactly in various Linux IO layers. blktrace shines here.

We reran the format with blktrace running. Here is what we got.


8,16 0 768 7.012762981 4916 Q W 6736 + 32 [istiod1]
8,16 0 769 7.012766958 4916 G W 6736 + 32 [istiod1]
8,16 0 770 7.012769506 4916 P R [istiod1]
8,16 0 771 7.012770511 4916 I W 6736 + 32 [istiod1]
8,16 0 772 7.012777259 4916 D W 6736 + 32 [istiod1]
8,16 0 773 7.012981610 4916 C W 6736 + 32 [0]
8,16 0 774 7.013543284 4916 Q R 6736 + 32 [istiod1]
8,16 0 775 7.013546301 4916 G R 6736 + 32 [istiod1]
8,16 0 776 7.013548430 4916 P R [istiod1]
8,16 0 777 7.013549233 4916 I R 6736 + 32 [istiod1]
8,16 0 778 7.014866251 0 UT R [swapper] 1
8,16 0 779 7.014880793 9 U R [kblockd/0] 1
8,16 0 780 7.014885958 9 D R 6736 + 32 [kblockd/0]
8,16 0 781 7.015813374 9 C R 6736 + 32 [0]

Till now it looks ok.
8,16 0 782 7.025277381 4915 Q W 6768 + 32 [istiod1]
8,16 0 783 7.025283850 4915 G W 6768 + 32 [istiod1]
8,16 0 784 7.025286799 4915 P R [istiod1]
8,16 0 785 7.025287794 4915 I W 6768 + 32 [istiod1]

A new write (W) request was inserted (Q) into the queue.

8,16 0 786 7.026059876 4915 Q R 6768 + 32 [istiod1]
8,16 0 787 7.026064451 4915 G R 6768 + 32 [istiod1]
8,16 0 788 7.026066369 4915 I R 6768 + 32 [istiod1]

Before we see the write request finishes, a new read (R) request was also inserted (Q) into the queue.

8,16 0 789 7.034883766 0 UT R [swapper] 2
8,16 0 790 7.034904284 9 U R [kblockd/0] 2

The block device IO queue was unplugged.
8,16 0 791 7.045272094 9 D R 6768 + 32 [kblockd/0]
8,16 0 792 7.045654039 9 C R 6768 + 32 [0]

Bingo! Here the read request was sent to device and finished before write.

8,16 0 793 7.045669809 9 D W 6768 + 32 [kblockd/0]
8,16 0 794 7.049840970 0 C W 6768 + 32 [0]

Repeat Linux mkfs and do not see such IO pattern. So because Windows and Linux do I/O differently during creating file system, such error does not show up in Linux. And since read the same block from last write request (especially bypass the initiator file system cache and send to storage directly) is so rare that we do not see error in normal I/O.

The root reason is because that version of blockio does not preserve the I/O order in all time. Andre, Ross, and I are working on that.

So with a tool that can tell you what precisely happen, life is much easier!

Leave a Comment

Older Posts »

Get every new post delivered to your Inbox.