[HN Gopher] Running a Unix-like OS on a home-built CPU with a ho...
___________________________________________________________________
Running a Unix-like OS on a home-built CPU with a home-built C
compiler (2020)
Author : markus_zhang
Score : 236 points
Date : 2022-11-27 08:30 UTC (14 hours ago)
(HTM) web link (fuel.edby.coffee)
(TXT) w3m dump (fuel.edby.coffee)
| IncRnd wrote:
| October 4, 2020, Running a Unix-like OS on a home-built CPU with
| a home-built C compiler
|
| https://news.ycombinator.com/item?id=24680109
| Tepix wrote:
| This would be a fantastic project to build a capture the flag
| challenge upon. Having to reverse engineer the platform to
| exploit it...
| [deleted]
| hardware2win wrote:
| Thats some high quality school
|
| Holy shit
|
| Im envy
| markus_zhang wrote:
| I'm thinking about getting nand2tetris done and go from there.
| None of the parts (build CPU using FPGA, bootstrapping compiler
| and porting an OS) seems to be particularly unreachable
| theoretically but of course they are all very difficult for me
| as a non CS guy.
| neverartful wrote:
| It's fantastic to read about experiences like this. What a
| tremendous level of accomplishment and satisfaction they must
| have felt! The closest thing that I've done to this was in taking
| 'Nand to Tetris' which is brilliant
| (https://www.nand2tetris.org).
| bo1024 wrote:
| Am I right that this is all simulated in software? I mean for the
| "implemented on FPGA" part, is there a physical FPGA running this
| OS? If so, how does it talk to the rest of the computer (i.e.
| drivers, etc)?
| magicalhippo wrote:
| The original Xv6 code uses the x86 IO ports for
| example[1][2][3], it would be easy to implement something
| similar for the soft CPU I imagine. Some good info about IO
| ports in the answers to this[4] SO question.
|
| [1]: https://github.com/mit-
| pdos/xv6-public/blob/eeb7b415dbcb12cc...
|
| [2]: https://github.com/mit-
| pdos/xv6-public/blob/eeb7b415dbcb12cc...
|
| [3]: https://www.felixcloutier.com/x86/out
|
| [4]: https://stackoverflow.com/questions/3215878/what-are-in-
| out-...
| aewens wrote:
| > A field-programmable gate array (FPGA) is an integrated
| circuit designed to be configured by a customer or a designer
| after manufacturing - hence the term field-programmable.
|
| > FPGAs contain an array of programmable logic blocks, and a
| hierarchy of reconfigurable interconnects allowing blocks to be
| wired together. Logic blocks can be configured to perform
| complex combinational functions, or act as simple logic gates
| like AND and XOR. In most FPGAs, logic blocks also include
| memory elements, which may be simple flip-flops or more
| complete blocks of memory.[1] Many FPGAs can be reprogrammed to
| implement different logic functions, allowing flexible
| reconfigurable computing as performed in computer software.
|
| The above being from the FPGA Wikipedia page[0].
|
| So yes, it is done "in software", but FPGAs is fancy hardware
| that can reprogrammed to act as new hardware on the fly using
| software. Many are implemented as PCI devices, so they could
| interact with the rest of the system over the PCI bus.
|
| [0] https://en.m.wikipedia.org/wiki/Field-
| programmable_gate_arra...
| bo1024 wrote:
| Sorry, my question wasn't clear. What I am asking is whether
| the FPGA was being simulated in a VM, or whether the OS they
| built was running on a physical FPGA. And if the latter, how
| did the OS handle drivers for other components.
| VLM wrote:
| Apparently simulated.
|
| You can see in the UART code for example:
|
| https://github.com/nyuichi/GAIA3/blob/master/hardware/Rx.vh
| d
|
| file input_file : ft open READ_MODE is input_filename;
|
| read(input_file, c);
|
| data <= std_logic_vector(to_unsigned(character'pos(c), 8));
|
| And so on and so forth, to pump a file on the simulation PC
| into a VHDL logic array one byte at a time as a simulation
| of a UART.
|
| Would be pretty funny if the above is the "wrong" repo for
| the story, but it is at least "an implementation" of the
| GAIA architecture, if not "the implementation" from the
| story.
| dingosity wrote:
| I used to joke that I was a "Full-Stack Engineer w/ VLSI Design
| Experience." But... I _never_ wrote my own C Compiler from
| scratch (or an OCaml implementation.)
| userbinator wrote:
| _The instructor only asks the students to "take this ray-tracing
| program written in OCaml and run it on your CPU implemented on an
| FPGA"_
|
| _However, some teams put more energy into doing fun such as
| running games or playing music by connecting a speaker with their
| CPU._
|
| That reminded me of this Linus Akesson demo, where he himself
| makes an entire SoC, including a GPU, and the software for it:
| https://www.linusakesson.net/scene/parallelogram/index.php
| dj_mc_merlin wrote:
| > CPU experiment is a little famous exercise held in the winter
| of the junior year in my department, the Department of
| Information Science at the University of Tokyo. In the
| experiment, students are divided into groups of four or five
| students. Each group designs an own CPU architecture, implements
| it on an FPGA, builds an OCaml subset compiler for that CPU, and
| then runs a given ray-tracing program on the CPU. Typically, one
| or two people are responsible for each of the CPU, FPU, CPU
| simulator and compiler.
|
| I have seen professional programmers fail projects much easier
| than this. How does the University of Tokyo manage to teach their
| students so well, or do they show up to it better
| trained/motivated than the average Western student?
| shakow wrote:
| We did something similar in my engineering school in France
| too.
|
| Design a simple SPARC-like CPU, make it work up to VGA/UART(and
| PS/2 IIRC?) I/O on an FPGA board, write a compiler for a tiny
| subset of Java for it, and make your OS work.
| srvmshr wrote:
| Alumni here.
|
| This is not a course everyone enrolls in. Generally the top 10%
| who are competitive enough enroll on it. But kids have done
| exceptional projects in this course. The creme de la creme
| students of Japan are in Tokyo U or Kyoto U, so the
| intellectual pool is generally very good (Tokyo University
| generally ranks within 40 consistently in CS/EE
| internationally)
| RMarcus wrote:
| Thanks for the first-hand information! I was also curious
| about this.
|
| > This is not a course everyone enrolls in.
|
| This is the ticket -- at both the University of Arizona and
| MIT, I've seen a group of folks graduate with a CS degree
| after taking OS, compilers, databases, and abstract algebra.
| Another another group of folks graduated after taking HCI,
| software engineering, design, and psychology courses. The two
| groups had some baseline skills (all knew the basic data
| structures and algorithms), but otherwise appeared quite
| distinct.
|
| I don't know how to phrase this formally, but I think some
| statement like the following is true: within-university
| variance is higher than between-university variance.
|
| (When I was younger, I had strong opinions on which one of
| these groups were "real" computer scientists. This was a very
| unfortunate way of thinking that prevented me from talking to
| folks who I later realized were some of the smartest around.
| I wish someone had corrected me sooner -- solving a problem
| with inputs/outputs well-defined enough to apply "rigorous"
| techniques doesn't make those problems inherently valuable or
| "harder" than others. God gave all the easy problems to the
| physicists.)
| pjmlp wrote:
| From this side of the atlantic that always seem quite
| strange to me, the only way to mix so disparate set of
| skills is to have two degrees, the software engineering
| degree of the first group + the HCI and psycology stuff in
| a second degree for social sciences.
| hnthrowaway0315 wrote:
| Thanks for sharing. Can you please share some short stories
| about those 10%? I'm curious what kind of education they
| received before they managed to pull that off. Judging from
| the article posted by OP, one gentleman was able to create a
| mini-curse "in 4 hours", which is impressive. I have a
| feeling that they are "Carmack-grade" programmers.
| trollied wrote:
| This same sort of thing was all taught on my degree course
| (Manchester, UK). Implement an ARM core (with a reduced
| instruction set), run on FPGA, write a compiler in the
| compilers course etc etc.
|
| It set me up well - I've written a few emulators as a hobbyist.
| neetfreak wrote:
| Got any materials on this? I wouldn't mind learning and
| attempting something similar.
| rwmj wrote:
| There are loads of free RISC-V cores that you can read the
| source of and run on cheap FPGAs. Take a look at PicoRV32:
| https://github.com/YosysHQ/picorv32
| matthewmacleod wrote:
| My CS course at Edinburgh had similar levels of projects, as
| did many others I've heard of - so I don't think it's anything
| particularly out of the ordinary for a good teaching
| university.
|
| The thing is, "professional programming" is a very different
| challenge from a task like this one, and it's hard to describe
| either as objectively "easier". The provision of a clear
| outcome, resources, guidance, teaching time etc. can go a long
| way to helping deliver a good outcome - often things
| professional developers don't have as much access to.
|
| I'd bet the majority of developers I know (and respect) put on
| to this project would fail at it without support - similarly,
| an early-career student would likewise crash and burn if put on
| to, say, developing a front-end architecture for a modern web
| app. Horses for courses.
| zmachinaz wrote:
| In order to get admitted to University of Tokyo, you have to be
| top on the common national university entrance examination, and
| then be top on the University of Tokyo entrance examination. So
| only the best of the best in Japan gets admitted. Highly
| competitive, as a degree from that University sets you for
| life.
| dingosity wrote:
| Just 'cause you're smart, it doesn't mean you know how to
| design CPUs (or write C Compilers.) And if the national
| university entrance exam is testing for VLSI design domain
| knowledge, they're doing something wrong.
|
| It sounds like UTokyo is like MIT. My first test there had
| material that wasn't in the book and wasn't covered in the
| lectures. When I asked my instructor about that, he said "Oh.
| You're supposed to know to do research outside of class," and
| when I asked my class-mates they said... "Oh. Yeah. My
| fraternity maintains a file of past tests for each
| instructor. You can get a good idea for what's going to be on
| various tests from reviewing the files. And if you don't
| understand it, you can get a frat brother to tutor you on
| it."
|
| (As an aside... one of the frats had THE EXACT SAME TEST I
| had just taken. The prof didn't even change the problems.
| _sigh_ )
|
| So... my experience with MIT was that it's a place where
| bright kids go to get taught by upper-classmen and the
| classes are there only to prove you have some sort of
| mastery, or at least familiarity, with the material.
|
| At Xmas that year I was bemoaning this fact to a friend from
| high school. His father was a physics prof at the local state
| university and overheard my complaints. He offered to give me
| a place in the lab if I xferred over. I took him up on his
| offer and wound up with a desk in the undergrad office, an
| account on the departmental VAX (this was a big thing back in
| the day) and a key to the physics building and the optics
| lab. I could drop in to my professor's offices virtually any
| time and most of them were excellent in explaining the finer
| points of quantum chromodynamics or math methods. I was
| recruited to be on the "let's build a super-cheap STM" team
| and landed a scholarship award for my work. After taking a
| hiatus to defend democracy I returned and landed a part-time
| gig at the Superconducting Super-Collider. That year I got my
| own MicroVAX. (Thank you, congress for all that SSC money.)
| But, of course, it didn't last. (Curse you, congress for
| taking all that SSC money away.)
|
| My point may be that smart kids will do well at whatever
| university they attend. Also, I think I did _MUCH_ better at
| the local state school than I would have at MIT. (Though I
| didn 't actually graduate w/ a Physics B.S. as I planned. IBM
| hired me before I graduated and it was a PAIN IN THE ** to
| matriculate with even a B.A.)
|
| Also. Lori Glaze (NASA director of Planetary Science) was a
| class-mate of mine, so... you know... it couldn't have been
| THAT bad of a school.
|
| Now... as a software development manager, the thing that
| REALLY impressed me was the ability of a group of kids to
| effectively work together. The United States pumps out a lot
| of very bright CS grads, and our national mythos of "the
| rugged individual" helps in some ways, but I'm going to guess
| most CS students aren't getting a lot of experience in groups
| larger than 2 or 3 people. When I hire recent grads from US
| schools, the most important thing we have to teach them is
| how to play well with others. Sounds like the UTokyo team
| already learned this lesson.
| laidoffamazon wrote:
| It's about being able to get in, getting out is irrelevant.
| People like me are a screwed for life because we didn't get
| in to any good schools in the first place.
| [deleted]
| lifthrasiir wrote:
| I have been in a university with that much competition
| (except that I enrolled back when the competition was less
| fierce, and I observed this as a teaching assistant), and it
| is very frequent that the competition only lasts until the
| point of matriculation because university curricula are more
| lenient in those countries. I would not expect most CS
| students in the U of Tokyo would be able to do the same.
| jcpst wrote:
| A lot of professional programmers, including myself, did not
| study computer science at a university.
| dj_mc_merlin wrote:
| Could you explain? How does not studying CS relate to being
| able to build a CPU/compiler/ray tracer? You don't need
| advanced CS to do so. Obviously the people who developed the
| main methods/algorithms knew their shit, academically
| speaking, but you can basically just engineer your way
| through all of this.
|
| Source: I also don't have a CS degree.
| jcpst wrote:
| > How does not studying CS relate to being able to build a
| CPU/compiler/ray tracer?
|
| I think it directly relates. University or not, I'm pretty
| sure you'll need to study some amount of computer science
| to do this.
|
| And then slightly related- When I was in college, I was
| able to deep dive into subjects like higher-limit just
| intonation and schenkerian analysis (music theory).
|
| So to what you're saying, I think you're right. When the
| parent of my original comment said "fail", really that's
| just "giving up", whether it be lack of time, lack of
| interest, or a lack of creative problem solving.
| dingosity wrote:
| When I went to college, most universities did not offer a
| CS degree. (Though the university I eventually went to had
| a "Computer Science and Engineering" degree that mixed bits
| of the Math Department's Computer Science concentration and
| the Engineering Department's Electrical Engineering
| curriculum.)
|
| I started my software career as a "that guy who can write
| FORTRAN" and eventually became interested in more computer-
| sciencey topics and less interested in particle physics. I
| wound up learning scheme by reading SICP and algorithm
| analysis by reading Knuth. I also lucked into a job where I
| got to talk to Ron Rivest every couple weeks, so probably
| got more access to him than the average MIT undergraduate.
|
| But mostly... getting a CS degree these days teaches you
| what to say during interviews. For instance, I stepped a
| kid through linked lists when I was interviewing for a
| position at Amazon. He had, of course, never heard of them
| and confidently informed me that hash-tables were the
| answer to every computer science interview question (I did
| not go to work for that team at Amazon.)
|
| As best I can tell, CS programs do a mix of teaching kids
| useful analytic skills and how to answer questions in
| interviews. I've hired both CS and non-CS degree holders.
| CS grads come with a pre-set "context" or "meta-mental-
| model" for how to solve computing problems. It is often
| very useful and I don't think you get that with other
| degree programs.
|
| However... in some corners of the world, that mental model
| is a detriment. I've had to untrain CS grads and get them
| to go back to first principals in some cases.
|
| I think the question is... is your organization doing
| something that would benefit from the contextual meta-model
| embedded in the CS curriculum at a typical university? Are
| you using Java? Are you programming server software on a
| posix-like operating system? How ambiguous are your
| requirements? And the EE kids get a bit more systems stuff,
| it seems. Half of the CS kids I've hired couldn't tell me
| how a larger or smaller cache would affect performance of a
| particular algorithm. All of the EE kids could. Managing
| ambiguity of requirements through judicious application of
| Gemma-esque design patterns seemed second nature to CS
| grads. The same concepts seemed to flummox EE grads.
|
| We are all victims of our training and the best thing you
| can say about someone who got an art history degree and
| then went into programming (assuming they can code) is they
| learned to analyze, design and implement software w/o the
| benefit of a conceptual scaffolding provided for them. They
| _COULD_ be the best coders for certain types of ill-defined
| problems; they 've demonstrated their ability to construct
| meta-models for evaluating real-world problems (again,
| assuming they can code.)
| robertlagrant wrote:
| > But mostly... getting a CS degree these days teaches
| you what to say during interviews. For instance, I
| stepped a kid through linked lists when I was
| interviewing for a position at Amazon. He had, of course,
| never heard of them and confidently informed me that
| hash-tables were the answer to every computer science
| interview question (I did not go to work for that team at
| Amazon.)
|
| My experience of comp sci and of FAANG questions lead me
| to believe that this is wrong (you do learn linked lists,
| even if you ignore them) and an outlier (FAANG interviews
| often have "implement a modified linked list" style
| questions.
| senkora wrote:
| These things are much easier for the average person when a
| course provides motivation, deadlines, requirements, and office
| hours.
|
| I would expect any Western student at a top school who self-
| selects into taking OS to be capable of this with the right
| course structure.
| dingosity wrote:
| I think that's related to how commenters are reacting. The OP
| sort of implies the only contact he had with instructors was
| at the beginning of the year when they received their
| assignment.
|
| I left a comment above about how I left MIT (which doesn't
| teach as much as it provides an infrastructure for taking
| tests students use to demonstrate mastery) for a state school
| (that actually taught the courses offered and provided
| students with ample lab opportunities.)
|
| I'm now thinking one of the great benefits from being taught
| by an actual professor (instead of an upper-classman
| fraternity brother) is that you get a context that is formed
| over the lifetime of a research career. I learned equations
| by reading the textbook. But how they were developed, why
| they were developed, what equations were used before and why
| they needed better models? I don't think you're going to get
| that from your fraternity brother.
|
| So... yes... hopefully the UTokyo team had access to the
| faculty to direct them towards the most fruitful areas of the
| academic garden. And yes... I think there are some very good
| CS students out there in the states that could do the same
| thing. But as I mentioned above, it's great to hear a tale
| about smart kids working together as a team.
| alfanick wrote:
| My experience is similar - Poznan, Poland - building your own
| CPU architecture is part of curriculum, most students hate it
| (and struggle to pass or don't pass because of it). For people
| who like it and get it, it's a perfect experience.
| iCurbix wrote:
| > building your own CPU architecture is part of curriculum
|
| I don't think it is any more. I did my BSc there and didn't
| have any course where you build CPU architecture of our own.
| extinctpotato wrote:
| In case of Poznan, it's the lecturer who fails to teach this
| subject in a sane way.
| secondcoming wrote:
| Do Polish universities not have an expectation that
| students attempt to figure stuff out on their own?
|
| Why do these students expect to happen when they enter the
| workforce?
| yarpen_z wrote:
| No, it's a different issue. I did my BSc in Poland, then
| MSc in Germany and now finishing PhD in Switzerland.
| There is a huge difference in the quality of teaching
| caused by decades of bad practices, lack of respect
| towards students, conservative mindset of faculty that
| prevents modernization effort, and lack of motivation
| caused by high teaching load and poor compensation. On
| top of that, you get the negative selection in Polish
| academia that results in filling positions with mediocre
| and passive researchers.
|
| Our students are not less independent or less inteligent
| than Germans and Swiss. It's our faculty and teaching
| staff that is often neither able nor willing to implement
| good teaching practices.
| dj_mc_merlin wrote:
| Can't speak for Poland but further east this extends to
| basic education too. The primary mode of education is
| memorization, teachers routinely insult pupils, and there
| is just a startling lack of understanding of how to
| enable a person to learn. They would much rather blame
| the student than look at their own methods.
|
| I have often wondered what causes this huge difference in
| efficacy of teaching between eastern and western Europe.
| Past influence from the Soviets? Wealth difference? Other
| cultural influences?
| markus_zhang wrote:
| But didn't USSR also produce some of the best minds in
| arts, science and technologies?
| [deleted]
| lifthrasiir wrote:
| When instructors were well motivated and designed the course
| appropriately, that can surely happen. I'm not sure if this
| particular course is mandatory though (see my sibling comment).
| pjmlp wrote:
| Most Portuguese universities have similar projects designing
| CPUs on digital circuits set of lectures.
|
| When I did my degree, it was many years ago, we still had a 5
| year degree for licentiate and this was spread between 2nd and
| 3rd year.
| tails4e wrote:
| This is an impressive project, and great example of teamwork.
| Yeas ago I'd used uclinux on an fpga, which was designed at the
| time to run on simple processors without mmu support, had you
| looked at that option? That said it's a lot more fun for the cpu
| guys to build those features and see them working. Any stats on
| the final HW design, mhz, resources, etc?
| gigatexal wrote:
| The ability to create almost everything I need for say a laptop
| or a desktop would be amazing. A custom built chip that could be
| 3D printed maybe -- maybe some derivative of RiscV will get us
| there sometime in my lifetime (next 50 years).
| digitallyfree wrote:
| You already can get part of the way there by desigining your
| own FPGA board (I believe there are some Lattice chips with an
| open-source toolchain, otherwise the toolchain is closed). On
| top of that you can run your RISC-V CPU and the other
| components you need to create a working SoC, with all the RTL
| under your control. Custom silicon in our hands would be quite
| a stretch though.
| robinsonb5 wrote:
| Yes indeed - the open source toolchain (yosys, nextpnr and
| GHDL for VHDL support) isn't 100% "there" yet, but it is good
| enough to use for real projects as long as you keep the
| limitations in mind. It supports several different FPGA
| families now, with Lattice ICE40 (project icestorm) and
| Lattice ECP5 (project trellis) being pretty mature, and
| (somewhat more experimentally) project XRay for Xilinx series
| 7 and project Mistral for Cyclone V - and others too.
|
| My own CPU (EightThirtyTwo) and SoC project (SDRAM, video,
| sound, interrupts, uart) can be built for ECP5 using Yosys
| and friends, as well as for a number Altera/Intel FPGAs using
| the proprietary Quartus.
| gigatexal wrote:
| This is a +1 to all those that commented below/above (I
| never know where these will get rendered). I had no idea
| you could already get pretty far. That's awesome.
| digitallyfree wrote:
| Glad to see that open toolchain support is improving, as
| FPGAs on modern nodes have the potential to run truly open
| hardware with reasonable performance. When I did my last
| FPGA project I was basically forced to use the Xilinx and
| Altera flows from start to finish. I work in the silicon
| industry and it's sad that everything is proprietary
| (compared to the number of open-source frameworks and tools
| on the software side).
| robinsonb5 wrote:
| Or maybe somewhat sooner?
| https://fosstodon.org/@mntmn@mastodon.social/109089458151612...
| VMtest wrote:
| since this is a university project, related to a comment on
| ryujinx emulator from a day ago
| https://news.ycombinator.com/item?id=33750368
___________________________________________________________________
(page generated 2022-11-27 23:00 UTC)