March 26, 2009

Re-creating old CPU designs

IMG_0036.jpg

Over the years I've done a number of experiments using Verilog, a hardware modeling language. In several of these experiments I have attempted to recreate old CPU designs like the MIT CADR lisp machine and the DEC PDP-8/I. My latest experiment is to recreate the PDP-11, in modern verlog, using modern simulation techniques.

Note that this has been done before. I know of at least 2-3 old microcoded versions and more recently there are 3 other groups which have done this, but in all the cases the code is either not in verilog or is proprietary and closed. Not very helpful.

I have not (yet) delved into SystemC, but I have done some fun work with co-simulation. Most recently I wired my RTL simulation of the pdp-11 in almost-verilog to a "known good" pdp-11 instruction set simulator. The idea is that both the RTL simulation and the instruction set simulator run the same code and at the end of each instruction cycle the results are compared. The "results" are the internal register values, the processor status word and the list of bus operations which occurred (address, type, data).

In a perfect world the two simulations will run in lock step and any deviation is a bug. And this is mostly true. The comparison turns out to be extremely helpful and very valuable.

Again, however, this is not new. I learned this technique from others who are smarter than I am.

While attempting to recreate the pdp-11 I ran into a number of interesting problems. The instruction set is fairly simple but it is not RISC. The effective address computations are complex and in many cases doubled. Let me supply an example:. Here is a list of the 8 addressing modes. A complex instruction can have a source operand (with one of these addressing modes) and a destination operand (with one of these addressing modes). So, in the worst case you need to compute the effective address and do one or more fetches for the source and destination.

mode symbol  ea1     ea2             ea3             data          side-effect                                                                               
0    R       x       x               x               R               x       
1    (R)     R       x               x               M[R]            x       
2    (R)+    R       X               x               M[R]            R<-R+2  
3    @(R)+   R       M[R]            x               M[M[R]]         R<-R+2  
4    -(R)    R-2     x               x               M[R-2]          R<-R-2  
5    @-(R)   R-2     M[R-2]          x               M[M[R-2]]       R<-R-2  
6    X(R)    PC      M[PC]+R         x               M[M[PC]+R]      x       
7    @X(R)   PC      M[PC]+R         M[M[PC]+R]      M[M[M[PC]+R]]   x       
Seems complex, yes? Each M[] is a memory read. The basic register indirect is simple. But modes 6 & 7 add the side effect of reading addition operand data from the next instruction location. This increments the pc as well as fetching an offset which gets added to the result of a previous EA calculation.

So, how to implement this? My first thought was a complex state machine. After a while I got frustrated and thought it might be easier just to make a machine which recodes the old pdp-11 instruction into new "risc-like" instructions on the fly. Sort of a just-in-time binary recompilation. I think this is how modern day X86 machines work. The fun idea would be to have several "machines" running ahead and converting the pdp-11 CISC instructions into simple RISC instructions, filling several FIFO's. The then RISC engine could use modern ideas like a multi-stage pipeline, speculative execution and branch prediction. While very cool, I quickly decided that was more complexity than I wanted at this stage.

I do think, however that it might make sense initially to do a simple "recoding engine" and a simple "risc pipeline". I want to do it and compare the gate count to a state machine version.

So, I set out to do a simple state machine version. I tried to compress the states as much as possible but current feel there has to be a decode state, four states for each operand, an execute state and a write-back state. The four states for each operand can be reduced to a little as one, depending on how the instruction decodes. I tried to eliminate the single EA state for each operand but instructions like:

   mov   @(R5)+,@(R5)+

causes problems. Why? because the value of R5 is incremented twice, once after each EA calculation. If I did the EA and post-increment in one state I needed to special case the increment (to be 2x) if the both registers were equal. And it got to be a big mess. I capitulated, added a state, and reduced the complexity.

I should note here that all pdp-11's, except one, are microcoded. And I can see why.

At some point I do want to try an experiment by adding a pre-fetch unit, keeping at least 3 words available and doing the EA calculations in parallel. The EA calculation will stack up (i.e. stall) queuing up for memory reads, but it has the potential for being more efficient, especially if there is a cache which does burst reads and the line size is at least 8 bytes.

I know this might all sound crazy, but I've learned a lot in the process and almost everything I have learned has been useful in my day job.

January 14, 2009

a used 2g iphone is actually cool

I bought a used iphone 2G model for work. I didn't intend to use it as an actual phone. But, as time wore on, I started playing with it, and (mostly) prying it out of the hands of my 10 and 12 year olds and I've grown to like it.

Oddly, I've yet to activate it. I suppose I will soon, but the idea of spending $75/month is a little painful right now. I guess I'm paying $50/month for my current phone, so maybe it's not that much of an increase.

The most fun part for me has been the mp3 player. I have a lot of music on our home file server, all legally paid for and ripped from CD's. Except for the mp3's I downloaded directly from Amazon (THANK YOU Amazon for selling MP3's!). Anyway, all of the music is legit. And now I can load it onto the phone and listen to it as I walk to and from work. Heh - I've become one of those "young people" who wear headphones. But I swear it's great to listen to music on the way to work.

I may have to upgrade to the 3G version, however, if only for the EDGE and GPS. I think the GPS might come in pretty handy.

I'm a little sad because my sleek little MP3 playing Sony Ericson slide phone is still pretty cool. I didn't intend to cheat. It just happened.

One thing I will mention - using iTunes can be a real pain. It's a very pretty interface but for the first time user it can be pretty confusing. I spent 15 minutes trying to figure out how to get music into it from a file on my hard disk. It should not be that hard.

I should also mention that the apps you can buy (or download for free) from the Apple "app store" are pretty cool. There's a lot of things to browese through. A few too many, to be honest. I would be nice to find some place the reviews the apps and recommends the best ones. Naturally my son (he's 10) immediately downloaded the Star Wars "light saber" app. Even I find it fun after a few coctails.

Even without activation the 802.11 connectivity make it a very useful device. I can browese, get weather, stock info, you tube, send email, etc...

All in all, I like it. And I didn't think I would.

January 8, 2009

pcc (portable c compiler) lives again!

Two interesting things happened this week

- the "R" programming language was talked about in the mainstream press.

- I discovered that the openbsd folks are working on using a non-gcc C
compiler (pcc). Turns out in the non-linux unix world there is not so
much love for gcc.

This makes some sense. gcc is huge and hard to work with. When it
compiles it uses every available resource and eats the machine. It's
not that easy to port or maintain. And it keeps getting bigger.

"pcc" on the other hand is 5-10 times faster, generates reasonable code
and is easy to port and work on. Someone is actually maintaining it.
This is the same pcc some will remember from Bell Labs in the '70s, back
when all the world was a pdp-11 (just before all the world became a
vax).

And, as eco systems go, it's good to have more than one option. Linux
is competely dependant on gcc. Netbsd on the other hand, is not.

(I've been working on getting netbsd to run on a my vax 11/730 again,
so I've fallen back in love with netbsd. Well, I think we're more like
friends with benefits, but please don't tell linux I've been cheating.)

anyway, I thought that was interesting. Apparently the pcc maintainer is
planning to add PIC support to pcc this winter which is one of the
missing features needed.

Ubuntu upgrades. wow!

I have a couple of machines running Ubuntu. More and more lately.

One machine at home was running Ubuntu 7.04 and mythtv. I was loath to change it
because it was working and I hate having to type "ssh" when I'm watching tv.

But, I finally did it over the holidays. First I upgraded to 7.10, which was a pain.
7.04 is not longer supported and I have to hack the apt config file to point to the
archives. But this broke the upgrade. So I ended up starting with a archive pointing apt
config and then switching it in the middle of the upgrade to point to the normal repository.
A bit hair raising but it worked.

Once at 7.10 I could cleanly upgrade to 8.04 and then to 8.10. I did it all via ssh and
and it worked with very few problems. My hat is off to the Ununtu guys.

Getting X to work consistantly with my Nvidia display card was no so easy. With some
releases there is support from Ubuntu. But not 8.10. For that I had to go back to using
the linux install script from NVidia, which does all sorts of fun things under the covers.

But now I have mythtv back, and I'm running 8.10 and all is well with the world.

That went so well I upgrade a machine at work and it too did the right thing. Very nice.

Sadly I'm about to wave goodbye to RedHat. I with I didn't have to,but they don't seem
to be providing the same level of coolness that Ubuntu us. The "apt-get" system is
just too nice. I never want to see another rpm again.

Virtually everything I wanted was available from apt-get. Amazing. And easy!

October 25, 2007

My experiment with Mythtv

I love my tivo. I've added a big disk to it. But I want to see more of my "personal media" on the tv (pictures, movies, etc) so I decided to make a Mythtv box.

I made a nice new pc in a "stereo cabinet like" box. I used a motherboard with built in HDMI output (very nice) and a 64 bit AMD cpu. I got all this from Mwave.

  • Motherboard: BIOSTAR TF7050-M2 nVIDIA GeFORCE 7050PV CHIPSET MICRO ATX
  • Box: ANTEC NSK2400 (BLACK / SILVER) MICRO ATX DESKTOP CASE

And of course I stuffed a dvd cdrom drive and a huge SATA disk drive.

Using a motherboard with HDMI out was key, becauase I just bought a big lcd hdtv.

When it arrived installed Ubuntu using this page:

MythTV_Feisty_Backend_Frontend

I bought a Tivo remote control because everyone in my house knows how to use one. I got it from WeaKnees. I love the stuff they sell.

I then found I had an "IR problem". I solved it with a USBUIRT device, which I also love. Get the one with the 56k detector. It "just worked".

Interesting bits:

- I found a tivo control file which described the tivo remote. And then I had to hack the mapping file quite a bit to get it to be the way I wanted.

- Support for the nvidia chipset was not in the kernel. I used an install package from Nvidia which was scary. But it did provide a nice X windows based config tool which turned out to be handy.

(X windows looks *really* nice on my hdtv. It makes me thing my next monitor for my office will be a 40" lcd hdtv. why not? huge screen!)

- DVD's did not "just play" by default. Instead I got odd /dev/hdc i/o errors. I asked a friend (thanks Bill!) and installed a few missing DVD decode packages. apt-get to the rescue. It then worked fine. I thought for a 1/2 day I had a bad cd-rom drive.

Result:

After many apt-gets and much twiddling I now have a nice Mythtv box which responds to a tivo remote. I can view my video and pictures and watch Jim Lehrer on the big screen (my record list is only PBS and F1 on Speed :-)

I may still get a HD Tivo, but the MythTv is a very nice adjunct and allows me to do things in a nice linuxy way with my home media. I plan to try firewire next and if that works ok I may skip the tivo hd...

March 18, 2007

Modifying read-only file systems in an embedded system

It's often a good idea to make the root file system in a embedded system read-only. If you do this and only make changes to files in a ram disk (mounted under /tmp, for example) the device will always come back to a known state when powered up. This is a nice feature and often a requirement.

But sometimes you need to make changes which persist. Check out the "mini fan out" file system. It allows you to layer changes on top of a read-only file system.

I have not used this package directly, but I plan to shortly.

Firefox bookmarks

Just a quick note on bookmarks. I use several laptops, a common machine at home and a workstation at work. This can sometimes get confusing when I save web bookmarks in various different places.

I recently discovered "foxmarks". I only used Firefox, and foxmarks is an extension which syncs up my bookmarks everyplace I install it. It's very handy.

I also discovered "gmarks", which places the bookmarks in a window on the left makes them easy to use. I'm still on the fence with gmarks

Sometimes I wish there was a way to make cross platform "notebooks" with firefox, which would combine a directory tree under SVN control, bookmarks, pdf and an email folder all into one. And make it easy to archive.

December 2, 2006

RCU in the linux kernel; an alternative to reader-writer locks

I ran into something in the announcement of the 2.6.19 kernel called "sleepable rcu". I found this wikipedia entry for rcu. It gives some nice background on rcu's.

I don't normally place much credence in wikipedia pages but this one seems reasonably good. And it explains why RCU's are a better alternative to multiple-reader-single-writer locks.

You don't care about this unless you're doing low level work on multiple CPU systems, which I do from time to time. I'm working on several at them moment, each small.

I'm new to rcu's but they look like a very low cost way to manage updates to high(er) contention data items which are updated (note the work "generations" in the wiki). This concept looks a lot like various academic techniques I've read about over they years to create safe software locks.

November 15, 2006

VPS - virtual linux at an ISP

Why VPS?

I had some problems with my colocated serve the other day so I thought I'd try out one of the "VPS" server companies. These are firms which sell "virtual linux", basically a virtual machine running on their hardware. It's a good idea if the hardware is located at a data center with backup power and multiple high speed internet connections.

But if you do some research you find out that many of these have a poor reputation. I wanted to find one one on the east coast which limited the field - many of the better ones are in southern California.

For the purposes of this discussion, "VPS" means a virtual server running Linux which makes use of Xen. It's being commercialized here.

Signing up for VPS

So, as an experiment, I signed up for three of them, vpsland, unixshell# and LeeWare Development.

All of them happily took my credit card and sent me automated email. But then nothing happened.

unixshell came back first with a machine - overnight - which is not bad. But the certificate on their billing web server is expired and 6 months old which does not fill me with confidence. I then followed their instructions to install a distro and boot. The install worked but the vm would not boot. And (apparently) no way to see why. I sent email. Nothing happened.

After 24 hours I opened a ticket on vpsland. Nothing happened.

LeeWare responded to email very quickly. They said they don't have any free Centos machine but I could use FC6. So, I agreed.

We'll see how this ends up. So far it's not a great picture.

1/2 a day passes
LeeWare came back first. I had to pick FC6 instead of Centos because they didn't have any machine free. But they did get the machine up and it seems fast and responsive. I've been installing things.

No word from vpsland or unixshell. My open tickets are unmodified.

(very late in the evening)
I got an email from a support person at unixshell describing all the problems they had setting up my account. They did finally get it going and it's working fine. Despite the late hour I start installing things.

(one night passes)
Over night someone a vpsland woke up and started setting up my account. This morning I could log in and start setting things up.

So, now all 3 are up and all within the promised "24-48 hours". Well, at least vpsland promised that. We'll see how the install/reboot/test cycle goes next.

Remembering the Original Goal

(several days pass)
Ok, everyone is up and I've gotten good at cloning my 4 services. Just to review, I am trying to run a web site with these services:

  • DNS - named as a master and slave to several domains
  • MAIL - qmail running as relay for several domains
  • WEB - apache 2.0 serving static pages and as a caching proxy to a little lisp based web server.
  • LIST - mailman running a very slow mailing list

I managed to clone my existing colocated server to 3 vps sites. All were up and running after a few days of fooling around. I was able to get all the services up with only a few problems.

Good things I found

  • yum
  • ssh, rsync
  • rebooting

Yum worked everywhere and was a very helpful. If something was not installed I just did "yum install xxx" and it worked. A breath of fresh air.

ssh and rsync worked everywhere. I made extensive use of rsync to clone whole directory subtrees. This turned out to be very helpful. I use rsync to clone my website - I keep all the files on a local server and then rsync to update the site. If I lost the entire VPS I would not loose my data.

After pairing down and configuring the servers, "/sbin/shutown -r now" worked as expected. The sites all seem reboot safe after a little tweaking.

Problems I found

  • Different versions of the base O/S.
  • Different versions of Apache
  • Selinux problems

Different versions of the base O/S

I wanted everyone to use Centos 4.4. But I ended up with FC6/Selinux, Centos 4.1 and Centos 4.4. For sanity I'd like to stick with Centos as yum works and it's a modern distro.

Different versions of Apache

Each site has a different version of Apache. And, the all behaved slightly differently with my proxy setup. I ended up building and older version (2.0.55) on all 3 machines since that worked. I've since changed my Apache config to rewrite the URL's and I bet that will work with the different Apache's - I'll try at some point.

Selinux problems

This turned out to be mostly fear of the unknown. I'd never used Selinux. It seems big and complex (because it is). All I wanted to do was run my old services. The biggest issues seemed be running my proxy server on a non-standard TCP port and moving my dns zone files.

Convincing selinux to allow me to run my server took some time to figure out but I found out there is a tool which will read the log file entries when it fails and generate config files to allow it to work. Moving my dns zone files from /var/named to /var/named/data was much easier.

I'm not sure I need selinux in the end, but it does make for a pretty bullet proof server.

Summary

As I said, I signed up for three of VPS servers. I managed to get them to take my money and set up a server. I was able to hack on the server, set it up the way I wanted and reboot it. It felt just like my existing colocated server, only faster.

Here's a summary of my experience with each ISP:

vpsland

Extremely slow customer support - almost non-existant. Took 1.5 days to get server up. Interesting billing web site which looks helpful. Very fast machine, very good connectivity. Located in Atlanta. Checkered reviews on the web. Flashy website. Good price/features.

unixshell

Better customer support - slow but informative. These are are clearly closer to the metal. Took a 1.5 days to get up but got some technical email about the issue (it's a bit odd that there was a problem since I ordered a vanilla vps, but I think this whole concept is a bit new). Somewhat fast machine, not bad. Ok connectivity. Located in Atlanta. Checkered reviews on the web. Techy web site. Good price/features.

LeeWare Development

Best customer support. I suspect this is a small shop but they give personal attention. The could not offer me Centos but did offer to refund my money and put me on a waiting list (which was very nice). Reasonably fast machine, good connectivity. Located in the mid-west. Very simple web site. Limited offerings but they matched what I needed.

Current thoughts

I like VPSland the best because they have Centros 4.4 and excellent connectivity. TThis is troubling to me, however, because their customer support is pretty non-existant. They do answer sales email, however, but only after 24 hours.

I like the unixshell# guys since I think they are more like me (geeks) rather than slick marketing folks. But their connectivity is not as good.

I really like Leeware because they were so helpful, but the lack of Centos (right now) puts them 3rd.

My current plan is to drop one and keep two. Originally I liked the idea of one in Atlanta and one in Chicago but right now I'm leaning toward ignoring the geography and focusing on two reasonable Centos based sites.

I totally expect one of these (or 2) do dissapear at some point. It's just too common an occurrence to ignore. So, my plan is to maintain two sites and keep them up to date. If one dies, I just move the DNS and don't look back.

The good news is that even two VPS servers are cheaper than my current colo server and I don't have to maintain the hardware. I just wonder how this will all look in 5 years.

November 12, 2006

HTML from lisp

(aside: this is a pretty funny intro to lisp casting spels in lisp)

Generating HTML from lisp is nothing new. Many people have done it and there are lots projects on the web using it. There's also a lot of interesting XML/HTML lisp code floating around. It turns out that XML and HTML lend themselves rather well to symbolic processing.

I've recently been frustrated writing and maintaining html web pages. So, I thought I'd try an experiment. I modified a simple scheme based http server, placed it behind a caching apache2 server. These are ideas I got while looking at Hunchentoot and TBNL. It seems you can make a pretty robust and quick web server with SBCL and these tools. I also like this package (htout).

My needs are more simplistic, and I'm lazy, so I used scheme (siod). I took my old html web pages and extracted out the boiler-plate from the content. The I wrote some translation code which would generate HTML from lisp s-expressions. This translation takes several steps as I implemented some simple macros to make my life easier (common things like anchors in side a list element for example).

It only took a few days and I had parity with my old pages and found the new "content only, lispy style" pages much easier to read and edit.

Here's an example of one of the pages from www.unlambda.com. This pages as a standard look and style sheet as well as standard navigation but all of that is hidden. I can use all of html, but I get to use nice s-expression which are much easier to navigate (for me) than the XML syntax.

;; $Id$
(project-page
 :title "Various Lisp Machine Keyboards"
 :start "Various Lisp Machine Keyboards"
 :body
 '(
   (p "These some pictures of various Lisp Machine keyboards.  I don't have
any documentation on the actually keys (yet).")

   (h2 "CADR")

   (ul
    (li-url "http://world.std.com/~jdostale/kbd/SpaceCadet.html"
	    "")
    (li-url "http://world.std.com/~jdostale/kbd/Knight.html"
	    ""))

   (h2 "Symbolics")

   (ul
    (li-url "/keyboards/symbolics.gif"
	    "")
    (li-url "/keyboards/DSC_2077-small.jpg"
	    "Symbolics 3600 keyboard")
    (li-url "/keyboards/keyboard_6074.jpg" 
	    "Symbolics keyboard (PN 365407 Rev C)"))

   (p)))

I'm pretty happy with this an plan to redo all my pages. Worst case I can always generate raw html and go back to the old way. Best case I can continue to leverage lisp and make better and more interesting (and active) pages.