With startling regularity students graduate from GWU without taking advantage of what DC has to offer. I wanted to help rectify this by at least making sure you know what opportunities await in DC! You can think of this as a bucket list, but realistically, your tastes will differ from mine, so use as you will.

Museums & Experiences

I’m not going to list the massive number of museums on the mall. They are all free, and you should check them all out! A shout out two two of note below as they are more interesting then they seem.

• The National Archives & the Library of Congress - Neither of these sound interesting, but they are both spectacular. I list there here as they are easy to miss and don’t sound that enticing. Both are free and very much worth a visit!

• Kennedy Center Millennium Stage - A free show every Friday! They have a massive variety of dance, acting, screenshows, and concerts, so keep your eye on what’s coming up. A simple 10 minute walk from GWU, so not to be missed!

  The top floors of the Kennedy Center are also fantastic as they overlook the river, Georgetown, and the Mall. A great “getaway” if you want an hour break to contemplate with beautiful views.

• Glenstone - An amazing modern art sculptural and experience garden spread over a massive acreage. The inside exhibits rotate, and the grounds stay large the same. Both are amazing, but I always go for the outside exhibits, especially “FOREST (for a thousand years…)” which is an audio experience in the middle of the forest. It takes at least an hour to walk around the grouds, so bring comfortable shoes. Free for students, and there is public transit (though it might take a while).

• Wolf Trap - A wonderful outdoor/indoor concert venue in warm months, and a renovated barn for concerts in the winter. Seating on “The Lawn” means that you bring your own picnic and drinks, and you get comfy to watch and listen to the music. Get there early to get a good spot; it can be hard to “squeeze in” late. No protection from the rain, so bring umbrellas, or watch the weather. My wife, friends, and I usually go here at least five times a year. There is public transit, but it is not super-close.

• The Dulles National Air and Space Museum - This is out by Dulles, but is well worth the trip. It is effectively two huge hangars attached, and filled with historical airplanes. There’s an SR-71, a Concorde, and a Space Shuttle. Anyone who appreciates the engineering or history behind these modern marvels should enjoy this.

• Dumbarton Oaks Gallery/Garden - The Garden is really the star of this show. It is a series of different, themed gardens spread around a large acreage. Not free, but well worth the visit.

• National Cathedral - The NC has amazing architecture, and a fantastic garden with meandering path. Interesting and beautiful even for the non-religeous/non-christian. The spectacular artificial cavern is amazing for organ and choral concerts.

• Staycations in Hotel Roof Pools - Many hotels in DC have pools on the roofs, and for a fee, you can hang out there all day! A great way to escape a little bit, get some pool and sun time.

Outdoors

Yes, we are in the middle of a big city. But there are plenty of opportunities to “get away” into nature.

• The C&O Canal trail - There is a trail that follows the canal in Georgetown, and then the river…all the way to Ohio. It is beautiful if you’re in the mood for a long walk, run, or bike. For much of it, there is a trail that follows the Canal that is dirt (thus easier on running legs), and a separate paved trail by the river (better for biking). A great example of feeling like you’re in the middle of nowhere, in the middle of DC.

• The Glover Archbold Trail - A trail you can get to from the C&O that turns into a small dirt path through a massive park. You can’t see the city around you at all, so it is a wonderful retreat from civilization.

• Dumbaron Oaks Park - A park that goes beyond the Garden that is free, and has a wonderful set of paths to wander around, hear a stream, and daydream. Relatively close to GWU, so worth a visit if you want some “nature time”. The Park is behind the “park” with open grass spaces, a kid play area, tennis courts, etc… Take the road down the hill to the DO Park.

• Great Falls - A beautiful park with overlooks of the massive set of waterfalls in NoVA (Northern VA) with a set of nice trails to get a little hike in. A drive away.

Coffee Shops

I’m a huge sucker for a good coffee shop. I’ve added distance location so that you can plan out the logistics of your caffeine-consumption plans.

• Baked and Wired - A 25 minute walk from GWU, in Georgetown. Great coffee and cupcakes. Their chaider is fantastic when in season. It is wonderful to take your goodies to the waterfront park. Very busy on the weekends.

• Bluestone Lane - A 7-10 minute walk from GWU. Very expensive, but has nice seating inside and out. Better than Starbucks, but if it were further away, it wouldn’t find a niche. This is the only chain on the list, mainly for convenience and the ability to sit down.

• Filter - A 10 minute walk from GWU. An interesting, small business coffee shop that diplomats commonly frequent. The diverse clientele makes people-watching quite interesting here. Good coffee, and well worth supporting. If you don’t want to sit down, this is better than Bluestone.

• Emissary - A 20 minute walk from GWU, and my favorite coffee shop in the area. An amazing place to work (when it isn’t busy), and fantastic coffee.

• Tryst - In Adams Morgan, so around a 40 minute walk. Adams Morgan is quirky, interesting, and full of great bars with lots of live jazz. Tryst matches the neighborhood. It is packed with old couches, significant diversity, and is, for me, the right level of dirty and disorganized to feel maximally gritty. My only sadness is not visiting this place more frequently.

Drinks

I don’t drink anymore, but these places really stood out when I did.

• The Gibson - I know that the speakeasy craze was over five years ago, but this place has always been fantastic. It is the type of place where you tell them the types of drinks you like, and they go crazy with creativity. I haven’t been here since Covid, so quality might have decreased (as it did in many restaurants). Get a reservation if you’re going. It is a hidden hole in the wall, so know where you’re going.

• The Sovereign - A Belgium restaurant with a strong selection of Absinthe. They have a proper Absinthe drip, so I strongly suggest that two people order it together. Its a whole thing. A nice environment (esp. downstairs) for hanging out.

Food

There’s a lot of distinctive food in DC, so I’m sticking to places I know, that stick out in some dimension.

• Kafe Leopold - A fantastic restaurant with amazing baked goods, and interesting drinks. Largely German menu, so expect a lot of kraut and sausages, but there is a fair amount of variety. More difficult place to eat if vegetarian, but the sides are wonderful. The baked goods, though lacking gluten-free options, are stellar – check out the glass case indoors. Inside and outside seating.

• Chaia - Wonderful vegetarian/vegan tacos. Relatively good value for money. The corn and carrot tacos are surprisingly amazing. You can eat inside (including upstairs), but we always sit by the canal or river and eat.

• District Tacos - Good, cheap tacos and burritos. A “goto” place for simple, filling food, done well. Relatively close to GWU.

• Bubbies - Amazing plant-based comfort food, only for pickup/delivery. Fantastic gluten-free options. The (gluten-free) onion rings are sensational. Not cheap, but worth the occasional hankering.

• Das Ethiopian Cuisine - DC is known for Ethiopian food, and Das is right around the corner! Worth trying if you aren’t opposed to some flavor. Not friendly (last I checked) for either gluten-free nor vegan (look out for the clarified butter!).

• Busboys and Poets - A DC staple with a great environment, and good food. Has lots of vegan/vegetarian/gluten free options.

I’m happy to add to this list if I missed anything, though I’d rather stick to locations I can vouch for. Post in the Discourse forum below!

In an ideal world, each PhD researcher¹ would enter into a department and into a relationship with their advisor that is centered on their continual development as researchers. This is not always the case. Like any relationship, the PhD researcher/advisor relationship can go very wrong. Often, it is hard to know when it has gone wrong, and what to do. I’d like to give my perspective on what students should rightfully expect from a PhD environment – a bill or rights of sorts – and on some thoughts on what to do if a student finds themselves in a toxic situation.

I am not a perfect advisor, and I have made many mistakes. Thus, this is only my opinion and perspective. I believe it is useful as this is a market with extreme information asymmetry, so I figure that having more information and options accessible is a good thing.

This is a very long post (~6000 words), so feel free to read the sections that are most interesting to you.

PhD Background

The goal of a PhD is to become a world-expert in a topic, create new knowledge in that area, and learn how to effectively communicate and disseminate that knowledge. PhDs often take between 4 and 8 years (depending an many factors) because it involves

• learning a deep field to the extent that you can extend it,
• knowledge creation is a necessarily creative process, which takes practice, context, and comfortability in your domain, and
• learning how to productive present your ideas takes lots of repetition.

Each of these activities are very personal. Each individual will approach them differently, and will most productively learn with specialized coaching. This is part of the reason that PhDs are based on the mentor system: an advisor leads the rising researcher through the salient work in a field, how to dive into new research problems, and advises them on how to present their work in a specific community. This is a relationship that serves as an effective means to enable to creation of world-class researchers.

Unfortunately, this relationship can go wrong. Most of academia is based on the assumption that professors are doing the right thing for students and advisees. Most of the time, the system works. However, there are core academic incentives that can negate this assumption, and there are bad people in every field that might take advantage of their students. When a PhD student finds themselves in a bad situation, it can be exceedingly difficult to know what to do, or to feel “boxed in” with no power. It is important to understand what to expect from an advisor/researcher relationship, if for no other reason than to know that the situation you’re in is often not your fault.

PhD Researcher Rights

What should every PhD research expect from their department, and their advisor? As a PhD student looking to complete their doctorate, I believe you should expect at least the following.

• Respect. To be treated as individuals with the respect of being on the hard path to becoming first-class researchers. As you learn about the depths of a domain, there is a phase where everyone starts to feel that they know very little about CS and about research. This is quite intimidating and can challenge your confidence. This can lead to slower progress than you may like as you wrestle with complexities and nuances. When you write papers, start designing systems, or start running experiments, it is hard to remain scientific (though this may help) and have confidence in your direction and decisions. This is all natural and part of learning new, challenging skills.

  Your advisor is there to lead you though this process, point out where things can be done in a different or better way, and explain the reasoning as to why this is the case so that you can learn and become increasingly capable. If you feel like your advisor is berating you, making feel fundamentally incapable or insufficient, or generally degrading you sense of self and your mental well-being, this is not part of the normal path of learning. They should be there to help you improve your capabilities, not shame you if there still are improvements to be made.

• Advisor as mentor, not boss. A research advisor is not a boss. A boss can tell you what to do within the broad parameters of your job, and what the company needs. A research advisor is a mentor - someone who should walk you through the skills, processes, and methodologies necessary to successfully conduct research in the chosen domain. If you don’t want to work on a project, you don’t have to. If you don’t want to make a deadline to protect your sanity, you don’t have to. If you don’t want to do work in a manner that fails meets your ethics or standards, you don’t have to. Advisors cannot tell you what to do, full stop. They can mentor you, collaborate with you, and convince you of the best direction to take.

  There are some trade-offs, of course. If you don’t want to take part in the commitments that an advisor has made (i.e. work on funded proposals), or want to work on topics outside of their interests, they have the option to not commit their time and resources to you. Our grants fund our research in specific areas, and we cannot pay for research outside of those areas. This might simply be a situation where you and your advisor are not a good match. This is not uncommon. In some sense, it is amazing that researchers can come together with similar interests as often as we do! Please see the section below on parting ways with your advisor.

• Choice and a threat-free environment. I know of cases where a professor coerced a student to do a task that they did not want to do, under the threat that their VISA will be terminated and they will be deported. This is obviously wrong, and is one of the best arguments I’ve heard that tenure is broken. I’ve heard of cases where the Doctorate degree was held as a means to coerce students into doing tasks that didn’t relate to their thesis. Threats – including those that are this obvious, and those that are subtle – have no place in academia. Threats are different from statements of fact that could take the form: “if we don’t make this deadline, we might get scooped”, or “if this result is correct, we need to re-evaluate the approach”. Facts should be taken at face value, and often propose a decision point (“do we all want to push for this deadline?”), or require additional investigation (“lets push into the results, and be willing to accept that our hypothesis was wrong”). Threats, on the other hand, often come with a lack of choice. If you feel like you’re put into a situation where you must do something, even if you don’t want to, then the problem is the situation and how you were put there. If you feel like you’re forced to fabricate results, or represent them in a misleading way, the problem is that you’re being forced to do that. If you feel like you’re forced to sink unreasonable amounts of time into work, the problem is that you’re being forced to do so.

  You should always feel that you have a choice, and that you’re free to make it without negative, malicious repercussions from your advisor. Your advisor should be there is there to support you, and sometimes it just isn’t reasonable make a push or get some results. The discussion should be about how to proceed after the decision, not about why you shouldn’t have the choice.

• Collaboration specifications. Collaboration on research should be respectful. You should feel confident that if you present an idea or a direction that it will be considered and seriously discussed. If it is not chosen as the direction of the research, you should know why, and see the reasoning behind the decision. If you feel like you are not being heard, or are only there to listen to your advisor tell you what to do, that is not a collaborative, mutually respectful environment. See my discussion of psychological safety for more detail.

  All researchers (including senior researchers) must collaborate appropriately with other researchers on a project. When you publish a paper, there is often a lead author (or small set of lead authors) that drive the research, a number of supporting authors that contribute to the research direction often through comparison cases, applying the research in a number of domains, and, generally, running additional experiments, and the senior researchers that at least mentored the process to completion. Different communities, and different research groups define these sets of researchers differently.

  All researchers in a collaboration should receive a benefit from that collaboration, and should have to contribute to it. See the ACM authorship guidelines for an example of how to think about this. This breaks in a few cases:

  • If authors are added to a paper not because the did work on the research, rather just to pad out their paper count, this is wrong. The decision if a researcher’s contribution is sufficient to be included on a paper is usually the call of the lead author, and this is a significant responsibility. Our names should only be associated with work for which we made a contribution. If someone does not contribute research to a paper, is included as an author, is later asked about the details, and they cannot clearly define the significant contribution you made, it will raise the question “what other papers did they not really contribute to?” If I see faculty applicants where all of their papers have tons of authors, I look at their group’s publications and that is the case for all of the papers, it raises the question to me if I can trust the author list as representative of contribution at all.
  • If a researcher does a lot of work that ends up being used without them being recognized, this can be a problem. I’ve heard of this happening in domains adjacent to CS quite a bit. In these cases, a researcher provides the computational foundation enabling some research and isn’t recognized as they didn’t contribute “directly” to the research. Before you contribute, you should clearly know if that contribution is sufficient to be a co-author. This should ideally be communicated directly to you. If you’re asked to some work, it is always appropriate to ask if the goal is for it to be included in a paper, and if that means you’ll be a co-author. Clearly communicating this key aspect of attribution is essential, and any ambiguity here is not OK. For example, “do this work and we’ll see if you can be a co-author” is not acceptable.

• Research group responsibilities. It is typical for PhD researchers to be expected to integrate into and contribute to a research group. This takes many different forms in many different groups. This can include helping run research discussion groups, helping more junior researchers, and providing assistance to other researchers in infrastructures that you know well. You should have a discussion with your advisor early on as to what these responsibilities are, and if you don’t want to take part in them that should be a discussion. The result of not wanting to take on some of these responsibilities might be that the research group is not a good fit for you. This is why you should have this discussion early on, so that you can pursue other opportunities.

  As an example of some of these responsibilities, in my group we try to focus on civil, productive discourse, and we’re all (me included) required to contribute to our shared software infrastructure. This means putting time into ensuring that our research results are shared in the infrastructure. The side effect of this is that each researcher is able to stand on the shoulders and contributions of past researchers. But it also means that time must be spent maturing the research implementations, and contributing them back to the infrastructure. If you’re in a group that really builds systems, this is a common requirement, but has the benefit that everyone in the group shares the contributions of that system.

• Expectations and parting ways. A PhD researcher should understand what is expected of them in terms of research, skill development, and research group involvement. Where the expectations aren’t clear, you should feel welcome to ask for clarifications or for a meeting to plan your goals. If you don’t meet expectations, you should still expect a supportive environment and approach from your advisor. Failure is frequent and necessary in research, and is hard to get booted up.

  If you don’t meet expectations over a long enough period, it might become evident that the research domain, your advisor, or some aspect of a PhD are not a good fit. Your advisor should have a discussion with you and clearly describe their thinking. In this case, you should still have the option to search out another advisor (inside or outside the department). This should really be the worst thing that your advisor can do to you: respectfully stating that they don’t believe the collaboration should continue. This must not be accompanied with threats, and they should be supportive of you moving on to your next endeavor.

• Development of skills. You should expect that your advisor provides you clear instruction of how you should develop your general skills as a researcher. This can include working with some infrastructure, practicing software development skills, presenting research, structuring your writing, and reading research papers. You should expect that your advisor will help with this by giving you opportunities for developing these skills, and provide input into how to manage your time.

  Presuming that you respect your advisor, you should take their advice earnestly, and devote the time to developing these skills. However, you should always understand why you’re devoting your time to a skill, and be convinced it will be useful for you as a researcher and professional.

  If there are additional skills that you believe you should have, you should feel welcome to ask your advisor for their input and, if they agree in the utility of the skills, how to acquire them. A PhD is about proving you’re a world leader in a domain which means devoting time to publishable research, but the skills you’ll acquire go well beyond that.

• Academic integrity. Academic communities are built up on trust. When you submit a research paper, it is peer reviewed, with the assumption that those reviews are unbiased and representative of a scientifically-grounded analysis of the research. If reviews are biased toward specific authors or institutions, the community suffers. Though reviewers do their best to analyze each paper, they usually do not attempt to reproduce the results, thus they rely on their intuition and experience to assess the validity of the results. There have been many stories about people gaming the system: friends providing biased reviews and accepting papers that might not stand on their own, papers that fabricate results to cover problems with the approach, etc.

  Fundamentally, academia should not be a pursuit of glory. It must be a pursuit of knowledge. If enough of a research community departs from this focus, then there is no reason to put any utility in the results of the community. Thus, many communities have in place mechanisms to protect themselves. These take the form of lists of researchers whose papers must be rejected and researchers that are not allowed to be on program committees (thus provide peer review). These mechanisms can actually complicate the decision that a PhD researcher must make when told to fabricate results – they might feel like either choice (acting against their advisor, or fabricating results) will result in professional failure. Even in these cases, there are only difficult options, but if you want to be part of an academic community, you cannot hurt it by publishing fake/misleading results.

• Time management freedom. You should expect to manage your own time. This means that you should not be beholden to a strict hourly schedule, but also that you must learn and exercise the skills of time management. This is subject to some exceptions as there are shared group meetings, and there must be a reasonable schedule overlap between researchers in a group. However, within these confines, everyone should work in a way that is maximally productive for them. If you are given strict hours to work, or hours per week to work, this is generally not OK (especially for high numerical values). Every researcher is expected to make progress, and should be trusted to do so at the maximum rate that balances their own life factors with their research.

• Basic timeline, environment, and schedule. A researcher should have the ability to know what technical hurdles they must pass, what the timeline around those hurdles is, what equipment and space is provided for the researcher, Importantly, a researcher should know when/if their advisor has a sabbatical in the near future, and what is the plan for continued advising. Some institutions have a formal process about sharing this information, but my sense is that this is somewhat rare. An example (unfortunately through google translate) shows how useful something like this can be.

Summary. The PhD research/advisor relationship should be built on mutual respect, and should be based fundamentally on dialogue and choice. A PhD researcher should never feel like they have to do something that they don’t want to do. An advisor should be a mentor focusing on enabling the success of their students as researchers.

This is my brain-dump of the expectations for a PhD researcher. Am I missing any? Let me know on Twitter or in the comments below.

What to do When it Goes Wrong

If you find yourself in a bad situation in which you’re being coerced, threatened, or generally not feeling supported, you have a number of options. I won’t argue that these options are easy, but it is very important to realize that you are not stuck.

Parting Ways with your Advisor

It is possible that as you learn more about the problem domain and research, your interests diverge from your advisor’s. This is a natural, and somewhat common occurrence. Talking to your advisor, and your department chair should inform you of the options. However, if you’re in an unhealthy situation with your advisor, you might feel trapped, and it is understandable you wouldn’t want to discuss it with them. You should not feel like parting ways with your advisor is going to negatively impact your future, nor that you don’t have other choices.

One option is to talk to your department chair. You can ask for confidentiality in your comments, but you might want to ask them under what conditions can they not respect that confidentiality. Threats and harassment often have to be reported, thus breaking that confidentiality. You can get advice on how to handle the situation, but if you’re in a toxic situation with your advisor, the goal of this meeting is to understand the procedure for seeking out another advisor.

If you don’t want to be matched with another faculty, or in the worst case, your department chair takes the side of the professor and nicely tells you to “work it out with the professor”, then you need other options. First, you should always feel free to apply to other universities. Remember that there is a September and a January start time, though departments tend to accept more in September. The best way to ensure that you have options here is to ensure that you work on your communication skills, and on your network. To keep your options open, I suggest meeting and talking to other academics at conferences, and doing internships. These activities will give you a broader network of academics that you can leverage if you want to look for something new.

To protect yourself in case things go wrong with your advisor, you should focus on your communication skills. If you stay within your department, you might be asked to perform teaching duties, at least in the interim till you find another advisor. You should ensure that you work on your communication skills to the extent where you can be trusted to run a lab. To effectively talk about your research with a broad variety of researchers at conferences, you should work on your communication skills and ability to convey key points at a high-level. When doing internships, you have to work tightly with researchers with varying backgrounds, thus your ability to communicate your ideas and understand theirs is paramount. In short, to ensure that you have the maximum flexibility in your future, you should make improving your communication skills a primary goal. I have some notes on this.

Lastly, realize that if you were able to get into a PhD program, and have some amount of training in a deep technical area, you are competitive in the job market. Many students realize that a PhD is not for them, and often find rewarding jobs in which they are successful. Even if your transition away from your advisor makes it impossible to remain in academia, there are a monumentally large number of interesting problems that need solving in industry. Don’t build up your own self image solely around getting a PhD. Focus on doing interesting work, and realize that you’ll likely find it outside of academia if you must.

Venues for Getting Help and Lodging Complaints

A number of venues² for getting advice, help, and lodging complaints.

• Your department chair/head. You can insist that you just want to get advice and be vague about the specific situation to get a read on if the chair will be receptive. If you provide specific instances of abuse, coercion, or threats, the chair is likely bound to act on them. Seeking help from other professors that have shown themselves to be student advocates is another option.
• Community management is performed by committees within the IEEE, ACM, and other professional organizations. If you feel pressured to perform unethical actions, these might be appropriate venues. IEEE Technical Committees (the committees that administer the community) often provide venues to get help. For example, using the architecture community as an example, the TCCA Student Advocates initiative provide help and advice to students who are experiencing difficulties (via email). Sig committees perform comparable services within ACM and provide initiatives, for example, like CARES. The concern here is that some of the members of these committees might be allies of the advisor. If the chair of the committee is “safe”, you can contact them, and share your concerns.
• Student unions are charged with maintaining the working standards for graduate students. If you have a student union, it is worth seeing what services they provide.
• If you’re considering suicide, you should call the suicide hotline at 1-800-273-8255.

Why Things Go Wrong in Relationships with Advisors

There are an innumerable number of problems with the academic system. Here I’m going to focus some of the pressures that cause the bad situations that many PhD researchers find themselves in, and on the power relationship that makes this problem is so hard to solve.

Common Reasons for Researcher/Advisor Breakdowns

• Being a Professor unfortunately does not imply having maturity or responsibility. A horrible, but common reason that a relationship can degenerate over time is simply that the Advisor does not believe the researcher is doing well, and that they won’t likely finish their PhD, but they don’t have the maturity to say so explicitly. Some people are very conflict adverse, and simply will avoid being truthful with the researcher. To avoid this situation, you’ll have to ask your advisor direct questions. “Am I on track to finish a PhD in X years?”, “What is the next goal, and what is the expected timeline?”, “What are the improvements you’d like to see from me to succeed?”. Sometimes you have to pull the information you want out of them. This is not necessarily comfortable, but can save you years of being “led on”.
• A problem arises if the advisor expects a specific rate of meetings, which doesn’t enable the productivity of the researcher. Some advisors expect frequent meetings (> 1 week), which means they expect to see consistent progress in small steps. If the researcher makes progress in bursts that are less frequent than the meetings, they will have little progress to show in many meetings, which can causing tension over time. In contrast, if the advisor prefers infrequent meetings (< 1 week), then they expect significantly more independence from the researcher. Some researchers prefer this independence, while others don’t have the tools to be able to consistently push forward their research without more frequent feedback. A researcher should understand the meeting expectations, or feel free to ask what they are. If the current schedule is not working for the researcher, they have to have a frank discussion about how they feel they could be more productive. In the worst case, the advisor will refuse to change the schedule, and the only recourse might be for the researcher to part ways.
• A researcher’s dissertation work is important, but in many ways its goal is to enable their next steps in life. Learning to do research and how to interact with a scientific community enables productive work in corporate research and in academia. Unfortunately, the goals as a researcher differ significantly depending on if they possibly want to go into academia. A researcher must have a record that exceeds the base requirements for a Doctorate if they want to be competitive on the academic market. It is still common for each open faculty position to receive 300-400 applicants. Alternatively, if an advisor assumes all of their researchers will follow in their own steps, and be a professor, they might have very high expectations for the researcher’s output. If a researcher does not want to take that path, then there will be a mismatch in the expectations for the “bar” that their dissertation must achieve. In contrast, if the researcher wants to go into academia, but the advisor doesn’t make them aware of the requirements, then they are being set-up for failure. As always, the researcher should be very clear about their own goals, and make sure that the advisor is on board.
• Advisor incentives don’t always match up well with those of a researcher attempting to earn a strong Doctorate. For example, an advisor can be focused primarily on getting and maintaining funding. This means that their emphasis is on convincing sponsors that their work is worthy of funding which can involve an extreme focus on demos (e.g. DARPA), industry deliverables (sometimes only tangentially research-based), or, generally, work that might not focus on research deliverables that are acceptable to academic communities. Alternatively, an advisor might focus only on software development which is a major time-sink for a researcher attempting to have impact in their research community. I’m personally sympathetic to this as my group does have a large focus on system building. A researcher should be very clear about what the goals and requirements are of a Doctorate, and ensure that there is a balance between the research requirements, and the other constraints that the advisor may have. Researchers should avoid falling into trap of optimizing only toward the goals of the advisor when they aren’t in line with the researcher’s own long-term goals. For example, it is possible that a researcher that focuses only on software will not have sufficient research to successfully defend their thesis. Researchers should be aware of the broader requirements within their department for earning an Doctorate, and be cognizant of what they need to be competitive on the job market after they graduate.
• A researcher has to balance a lot of responsibilities that possibly include teaching (if they are a TA), classwork, and research group service (e.g. running group research meetings). It is very easy for an advisor to lose context on how much time a researcher is spending on these duties. This can lead to significant Advisor disappointment with the researcher’s rate of research progress. It is important that the researcher inform the Advisor how much time is spent on which tasks, and the best way to do this is often to let them know what your load outside of research will be the week before. This makes the discussion informative rather than possibly sounding like excuses. One of the most difficult parts of being a PhD researcher is balancing other responsibilities with making constant research progress. You do have to feel like your advancing as a researcher, and if you’re over-burdened with other responsibilities, you must have a discussion with your Advisor. They should be there to help you figure out how to constantly improve as a researcher.

Academic Incentives

Some of the bad situations in academia happen because the underlying incentive structures encourage researchers to focus on quantity of publication. It is hard to evaluate a researcher’s work based on a qualitative evaluation (though it is necessary). If they aren’t in your field, how do you determine the quality, trajectory, and potential of their work? In reality, you can do a decent job at this evaluation, but it takes time and effort. What’s much easier than this? Counting publications. This factors into annual salary increases, department rankings, job offers, tenure decisions, acquiring grants, etc… Thus, there is an incentive to publish more, even if no-one reads the papers.

The side effects of this incentive system are varied. They include:

• the widespread creation of many conferences and journals that are almost never read,
• the focus by some on gaming the metrics including wide-spread (inappropriate) self-citation, incremental research in search of the Least Publishable Unit (LPU), and a friends-helping-friends relationship in program committees, and
• the focus of researchers on the (somewhat perversely named) productivity metric.

Most debilitating, I believe this focuses research on getting papers accepted, rather than on doing interesting research. Though accepted papers are often on interesting research, I firmly believe that motivations matter.

Put simply (quote):

Above all, quality and impact need to be incentivized over quantity. Sheer numbers of publications (or derivative bibliometrics) should not be a primary basis for hiring or promotion, because this does not encourage researchers to optimize for quality or impact.

Reasonable arguments have been made that in the world we live, you must focus on both quality and quantity. I certainly agree that the world does value quantity in addition to quality, and you cannot rely on those who evaluate you properly focusing (or being able to focus) on quality.

If professors focus primarily on continual publication as their primary goal, it is natural that pressure trickles down to PhD researchers. In this case, the entire focus is on deadlines and on getting results sufficient for a publication. In the LPU mindset, it is not useful to get results that are stronger than are necessary for acceptance, as you might as well use the additional results to get the next publication accepted. This dilutes the quality of each publication, and I believe this is wrong to the core. Not everyone agrees. Thus, it is important for a PhD researcher to ascertain what the core values of their lab are, and what the advisor values in research.

Student/Professor Power Relationships

Students can easily feel trapped in a bad situation with their advisor. Many international students feel that the status of their VISA is under their advisor’s control. In the worst case, I’ve heard of advisor making physical threats. That the term “academic slavery” is something that many can recognize, if not identify with, is a sad indication of the situation.

A major issue is the advisor’s perceived lack of accountability, thus an inability of the PhD researcher to lodge a reasonable and serious complaint. If a PhD researcher believes that the department chair is unlikely to intercede, and that there aren’t outlets for complaint within the community, it is hard to see many options. Harmful advisors never need to change their behavior, thus will exploit PhD researchers across many academic cohorts.

At its core, academia is based on trust and responsibility. Thus professors are granted a large amount of independence, one of the main benefits of the job. Professor’s labs are similar to startups: small groups of individuals, aggressively pushing to get their technology adopted, while seeking a constant stream of funding. Professors are in charge of their research direction, and how they run their labs. This is by necessity as different research domains and areas of expertise require different directions and management. Further, they are supposed to co-manage the University and ensure it maintains appropriate research and educational standards. This leads to very shallow management hierarchies in which Professors are accountable to few. Professor’s management style and mentoring are under-evaluated during Tenure evaluations, but red-flags will be considered.

Regardless this independence, coercion, threats, sexual harassment, and the like, are incidents for which a Professor can suffer significant consequences. Unfortunately, the history of persecutions for such behavior leaves a lot to be desired. Regardless, PhD researchers in bad relationships are often less concerned with repercussions on their advisor, and more-so getting out of the negative situation. See the “Parting Ways” section above.

In short, professors have a lot of power over PhD researchers, and are often not that accountable. It is important to join a lab that has a healthy dynamic, and be willing to part ways if it isn’t a good match, or if the relationship is toxic.

Structurally Fixing Academia

In this post, I’m attempting to provide my perspective, given the current state of (American) academia. I also cannot more strongly recommend a number of current social movements including:

• the creation of graduate student unions where the union can protect individuals and represent the interests of all students,
• a wholesale switch over to the CRA’s recommendations on academic evaluation based on using a fixed and small (e.g. three to five) number of publications to discourage publication quantity, and
• support networks in communities that are distinct from any specific advisor.

I don’t think that these are sufficient, but they are large steps in the right direction. Regardless, my focus here is not on how academia has to change (it does), but on what a PhD researcher should expect, and what do to if they are in a bad situation.

What to Consider When Applying for a PhD

When you’re applying a position in a research group, you have almost no information about what academic life will hold, nor what the relationship will be like with your advisor. I want to provide a list of questions that might get you more information about the expectations and relationships of your prospective advisor.

• What are the responsibilities for each PhD student within the group? For example, does each PhD student mentor junior researchers, are students expected to contribute to a software infrastructure, and what are the student responsibilities for group meetings? This is not a common question, so don’t expect a concise answer.
• How is the author list of a paper decided, and who makes that decision?
• How many researchers are generally on each paper and what contributions do each of them make?
• How are the projects that each researcher works on determined?
• What is the schedule of a typical PhD researcher working with the advisor? What is a typical time commitment?
• What does the lead up to a paper deadline look like?
• How do you determine the set of evaluations for a paper?

It would be good to ask these questions to the current PhD students of the advisor, and compare the answers to that of the professor. In addition, you might ask the students:

• What are your typical meetings with the professor like?
• What pressure do you feel from the professor?
• What skills has the professor helped you acquire?

If you’re an international student, do not feel like you have to work with a professor from the same country or region as you. Diversity is a core tenant of academia, and it is important to learn from various perspectives. Immersing yourself in a group from a different ethnicity and background will push you to vastly improve your communication skills.

Perspective: When I Messed Up Advising

There are quite a few times where I messed up as an advisor. These mistakes have made me reflect about the root problems, and clarify my parameters for interacting with PhD researchers. I share these mainly as examples of what I see as a legitimate set of problems due to a Professor making mistakes and struggling with constant improvement. I still make mistakes with an alarming frequency.

• I am a very direct person, especially when it comes to research. I constantly worry about how direct statements are interpreted, and there is a very real danger that they are taken as a value judgment on the researcher. For example, if a researcher presents results to me, the first comments are usually of the form “if I understand this correctly, these are very bad results”. Over time, I’ve learned to catch myself, and qualify: “…great job getting these results, but I think that we need to dive into X to understand why we’re seeing the Y phenomenon”. Directness can work against the perception of respect and a supportive environment; but it is also necessary when productive discussing designs and results. I’m always working to edit myself to try and have the appropriate balance.
• A researcher was resistant to adding a research software project to the shared group github account. We need a shared infrastructure so that multiple researchers can contribute, but I failed to articulate how important this was before the software was created. They wanted the repository in their account for their resume, but I knew that I could promote the software and the researcher better if we could all improve and publicize the work (it has been used since by industry). This turned into a somewhat contentious discussion, all because I didn’t clearly articulate the group’s focus on shared software infrastructure.
• A similar issue arose as I, again, didn’t properly articulate the group’s goal of building an infrastructure from which we can all benefit (within reason). A researcher was turned off from previous experiences trying to fit into the confines of the system, and proposed doing research with zero contributions going back to the infrastructure from which they’d benefit. The core issue was that they didn’t want to learn the conventions of a (particularly messy) part of the system. The compromise was to contribute to the rest. My lack of communication about the importance of the shared infrastructure, again, led to mismatches in expectations.
• My largest mistake that I continue to struggle with is that I don’t properly prepare researchers in how to meticulously approach writing. Deadlines always put too much pressure on the writing process and result in me providing too much of the text of the papers. I’ve taken steps to mitigate this, but I am still not providing enough feedback and iteration for students to be self-sufficient in writing papers.

These shortcomings indicate that Professors are not perfect in managing their relationships. However, each of these have led to discussions about how the situation can be resolved for both me and the researcher. Importantly, I’ve reflected on how I can do better in the future.

I believe these are the scale of problems that are somewhat typical in academia. Professors aren’t trained in management, and we do a lot of learning on the job. However, problems with coercion, threats, and inappropriate pressure that removes PhD researcher choice, are not appropriate. I hope this document has provided some insight into the options that one has in these situations.

Updates and Edits

• Thanks to Sean McBride: Fixed the information asymmetry link.
• Thanks to Samy Bahra: Upgraded “researcher” to plural.
• Thanks to Paul Khuong: Added the bullet on the basic expectations and a link to a translation of a document used to spell out the basic expectations and the parameters of the researcher/advisor relationship.
• Thanks to Sam Tobin-Hochstadt: Added the section on causes for common relationship breakdowns. The topics of this list are derived from Sam’s great suggestions.

An important and necessary part of learning to do research is learning how to effectively convey your ideas to others. One of the main mechanisms we use for this is academic papers, and each PhD student spends a lot of time learning the conventions of writing a paper in their area, and learning the effective means of convey to others and convincing them of their ideas. Learning to effectively write papers is one of the important skills that you will acquire on the path to being an expert in your field, and receiving a Doctorate. You should go into the process taking this skill seriously, and with a commitment to master it. Especially if you believe you’re interested in working in research (academia, or lab), this is a required skill. This post discusses one version of the process for going from idea, to published paper. There are many different successful models for this, so look at this as a default option from which you can deviate with intention.

Idea \(\to\) Outline

Presumably you’ve been working with your advisor on the technical aspects of your work. However, when discussing and working on the “how” and “what” of your research, you should also back away and make sure not to lose the “why”. This section discusses the process to mature your research idea into something that can be the basis for your paper. You’ll find that the first time you do this, it will take a lot of time, and you’ll need to work with your advisor a lot to flesh it out. As you progress, and have done a few papers, this process is still hard, but becomes more methodical and efficient.

The goal of the outline is to give you some guidance for

1. what the fundamental motivation for the work is,
2. how it can be justified to the world,
3. what experiments make the strongest argument for the work, and
4. to enable you and your collaborators to be on the same page for the rest of the research and paper.

Make an outline.md¹ document in your paper’s directory in the publications/ repository (remember: making the paper’s directory makes you choose a conference and deadline). In it, focus separately on four main sections:

1. the motivation with a focus on key aspects of current systems that are insufficient,
2. the story, or the logic surrounding a justification for your research,
3. the contributions, stated tersely, and
4. the evaluation, which includes a sub-section per experiment, and a focus on the key questions each evaluation is answering.

The motivation and the story are related, but I break them into separate concerns because I’ve noticed that researchers have a tendency to not pay sufficient attention to the shortcomings of current systems. The motivation zeros in on this aspect, while the story provides the surrounding context that captures the logical argument for the work.

While you’re writing an outline, please ensure that your text is grammatical, and makes sense. This is the time to focus on terseness, and cutting to the core issue. If you’re writing long-form paragraphs, you must do a few editing passes to cut to the core of the argument.

The outline should include a section for each of these four sections.

Motivation

Focus on answering two questions:

• What is wrong with current systems, why does this motivate your design?
• What experiments would clearly show the deficiency of current systems?

This might require a description of existing systems, to provide context. You should try and make sure your motivation is to the point and as concise as possible. If it isn’t clear from the description, this certainly means that you have to understand other systems to be able to properly characterize them.

Try and succinctly summarize a small number of experiments or qualitative factors (max 2-3) that summarize what is wrong with current systems, and why that motivates your design.

Story

What is the arc of the story for the paper? How do you go from

• factors that people care about (speed, parallelism, costs, reliability) that are properly justified, to
• an argument that existing systems aren’t optimal for these factors (this might be redundant with motivation), to
• a value proposition for your work that it can improve systems for those factors, and finally to
• a justification for your rough design that is simple and can be summarized in a small number of bullet points (e.g. four).

This will get translated into the introduction and motivation sections, and will impact how you organize the description of the design, and the evaluation. It is important to make sure that your story is clear and as concise as possible. If you end up writing a ton of text, then make sure to go through it again, and simplify. I find it is often best to simply have your story be a bullet-list of items that logically take you through the story as this forces you to be a little more terse.

If your motivation and story don’t fit onto a page when spelled out tersely, then there is likely little hope they can fit into a paper, so focus on being concise. This focus will pay off when you’re writing the paper, and already know what the most important aspects on which you need to focus.

Contributions

Now that you’ve tersely stated the motivation and story, you should be prepared to equally tersely state the contributions of the work. The list of contributions is very important for a number of reasons:

1. it is your stated list of advances to human knowledge and capability,
2. it is what your audience should walk away from your paper with understanding and agreement,
3. it sets the stage for your evaluation which is mean to validate those contributions.

When reviewing a paper, the main sequence of questions I often ask are

• What are the contributions of the paper?
• Are these contributions compelling and of sufficient interest for the bar of the conference?
• Are these contributions accurate compared to related and previous research in the publication record?
• Does the design accurately encompass these contributions?
• Does the evaluation properly and completely validate these contributions (within reason)?

They are, in many ways, the skeleton that binds the paper together. Unfortunately, I do find it is hard to come up with them without properly diving into the motivation and story first.

No contribution should be more than three sentences, and you should likely not have more than five in total. Some “scoping” of contributions is required, and can be discussed with your collaborators.

Evaluation

This is the section that is often the most difficult and will consume the majority of your time and of the outline’s space. The point of this in the outline is to flesh out a lot of the details:

• What are the results, and how are they depicted?
• What systems are being compared?
• What are the relevant aspects of their setup and how are they configured?
• What are the structures of any tables, and the x- and y-axis, along with the different lines/bars for each graph.

If you have a “dimension problem” where too many variables exist in the system, this section will force you to figure out a way to represent an abstraction of those. Getting to this level of detail is difficult, and requires aiming to finish this outline two months before the deadline. Most importantly, the outline will give you strong guidance for what results you’re shooting for, thus give you good input into how you should spend your time.

Create a subsection within “Experiments” for each of the classes of experiments. There are often at least:

• Motivation - why are current systems insufficient? (perhaps this is redundant with your previous work on the work’s motivations.) Please start out with the motivation asking the question, “what do existing systems lack, or what is wrong with existing systems?”, and from there create experiments that fairly demonstrate those shortcomings. First and foremost, you’re trying answer the question “what value does your system add”, and focus on results that fairly showcase that. Only after you establish value, do you look at experiments to characterize all aspects of the system (where it is weak, where it is strong, workload characterizations, etc…).
• Applications - what higher-level applications can benefit from your system? These are important as they show the “final” justification for your work – the “proof’s in the pudding”. If your system doesn’t work for applications, then it will be difficult for your audience to completely believe your work’s value.
• Parameter studies - what are the parameters that impact the benefit or the properties of your system? For example, things like the working set, rate of arrivals, size of requests, memory allocations, etc… These are often the easiest to figure out as they are very much intertwined with the technical details you’ve spent months working on!
• Microbenchmarks - these study the atomic costs of the system with the intent of giving the reader an understanding of the underlying bounds of the system and compare, where possible, against comparable operations on existing systems.

Please not that this is not the order this often appear in the paper. Often the most natural order in a paper is motivation, microbenchmarks, parameter studies, then applications. However, for the outline, I’d like you to focus on this order as it forces you to consider the most difficult, first².

Specific Experiments

Create a sub-sub-section (that’s three “###”s) within each of the types of experiments, for each experiment. The first thing you should do is understand and write down the question you’re trying to answer with the experiment. For example:

Question: What is the maximum interference on high-priority tasks that systems can experience from IPIs?

Only after you clearly identify what the experiment is answering, should you then list the following:

• System setups - which systems are you comparing, how are they set up?
• Workloads - what is the workload on each system?
• Graph/Table details - the axis and parameters of the graphs/tables. This is the hardest part, and will take a lot of thought. The dominant question is how to best represent the phenomenon you’re studying, and how to best answer the question (from above).

If you have trouble putting any of this together, get your advisor involved, and talk to your peers! You’ll iterate on this document quite a bit, so don’t get too attached to anything. The story of a paper can change quite a bit as you get results, but you have to have an initial plan, and a goal.

The Headline Result

The headline result is a single graph that, alone, both shows why other systems are insufficient, and why your system is necessary. From another perspective, if you gave a presentation of your work to a company, which result would you show? This is something that I went for over ten years in ignorance, not knowing how important this was. The contrast to having a headline result is to have to go through many of your results to convince people of the contribution. For scientific completeness, we require a thorough evaluation, but if someone trusted that you did your research in an intellectually honest way, the headline result would almost be sufficient.

So the last thing to consider is which of these results is your headline? If it is hard to identify it, it is worth thinking about if they can be reconfigured to bring one out.

Collaboration and First Draft

After a sufficient number of rounds of discussion and revision with your collaborators, it is time to make the real paper’s outline. Keep the outline.md around, and don’t update it with changes in the real paper. This will be useful for a post-mortem for yourself (i.e. what did you do wrong in the outline phase?). Your initial outline should include sections, subsections, figures (often not populated at this point, but with descriptions)³. At this point, you should be very intentional with your collaborators, and come up with a plan for who is going to work on what, when. The goal should be to get a first draft of the paper done a couple of weeks before the deadline. This will give some time to send the paper out to friendly researchers for feedback, and time to iterate⁴.

Figures

The text is often wrapped around, and driven by the figures, especially in the design sections. Thus, it is important to plan the figures before diving into the text.

Headline figures:

• Motivational figures. What figures can capture at a high-level the difference between your system and existing systems? This might a number of shapes: a graph with N dimensions, showing points for different systems (e.g. scalability, consistency models, isolation modes) (see the BI paper); a diagram showing different system organizations, and implying different isolation and performance characteristics (see the TCap paper); or a “context” diagram showing how the systems fit into a broader context (the system in hardware, the system in an IoT environment, etc…). These are hard, and very important. They will capture the audience very early on in the introduction, and help transition into the system design. Note that some papers don’t have a natural motivational figure, but put effort into coming up with one.
• High-level design figures. If you had to give a lecture on your system, and you had one figure to use to guide the discussion, what would it be? Is there a way to summarize the design of the system in a single figure? When reading through papers, ask yourself which are the headline figures. They often aren’t hard to find.

Aside from these, you will likely require figures to explain different subsystems, and experimental setups. Take some time to think this through, and what additional figures are necessary. Add them into the outline (with at least a description).

Paper Organization Options

There are nearly endless options for how to organize a paper. There are a few high-level things you want to focus on:

1. Separate out the story of your work from the system design.
2. Separate the description of the high-level design of your work, from the implementation details.
3. Figure out how to properly justify the work (story), and integrate the motivation.

Paper organization should be done intentionally to manage the intellectual dependencies inherent in your research. These dependencies capture the ideas that are necessary to introduce a new idea. You can’t explain a new IPC mechanism without a previous introduction to protection domains, or to the architectures and shortcomings of existing approaches. You can’t describe a new RTOS without introducing the concepts of predictability and interrupt management. At a finer granularity, you can’t introduce TCaps without first introducing user-level scheduling, interrupt abstraction, reservations, and IPC⁵. The coarse-grained organization of a paper gives some indication of how these dependencies are managed.

Here are three organizations that we’ve used that all differ on the last point:

Conventional: This is what I consider a more “classical” organization that relies on a strong intro to motivate the work. The design section/s is/are separated from implementation to explicitly manage dependencies. In design, you introduce the high-level ideas behind your system, while the implementation includes details that often require an understanding of the high-level architecture. The related work is after all of the paper’s details, so you can leverage the full intellectual description of your system, and of the results in relating your work to other’s.

• Introduction
• Design
• Implementation
• Evaluation
• Related Work
• Conclusions

Note that in all options, Design and Implementation might have more research-specific names, and likely have many subsections and figures.

Motivation Section: This organization has an explicit Motivation section and might be appropriate if you have strong results that can go into the motivation to justify the need for your system before you’ve introduced the system. Note that the motivation section has very few intellectual dependencies it can rely on, so it has to introduce the motivation for your system, without any detailed descriptions of your system. This means that the motivation is often primarily based on quantitative properties of existing systems.

• Introduction
• Motivation
• Design
• Implementation
• Evaluation
• Related Work
• Conclusions

Related Work as Motivation: The related work section can be placed early in the paper where each previous system can be used as a foil to introduce high-level aspects of your system. From my perspective, this is the most difficult to write, generally, but some people do it very well. This is because the related work has to place your work with respect to other’s while relying only on the high-level description of your work from the introduction.

• Introduction
• Related Work
• Design
• Implementation
• Evaluation
• Conclusions

I’m using monolithic bullets for each of these sections. Each of them often requires an intentional break-down into smaller constituent parts. A few examples follow:

Notes on Design. The design section of a paper is very important as it lays out the abstractions and means for the reader to think about your work. I find it helpful to think about this section as constituting at least two parts: 1. the design goals that enable your design to achieve your contributions, and 2. your design that achieves those goals.

Notes on Evaluation. Your evaluation must include an analysis of the results. You want to cleanly decouple a description of the evaluation setup and what the graphs show, and how that should be interpreted. In the past, we’ve done this by adding a Discussion. heading for a paragraph after the experiment’s description. I’ve read many successful papers that have added subsections that summarize key take-aways from results as well. Regardless, make sure that you intentionally separate description from interpretation.

Writing the Paper

I’m not going to go into this now, but you can find some links below that are decent. I’ll be involved in this process, so never hesitate to ping me if you have doubts or concerns.

Throw This All Away

Once you feel comfortable navigating the difficult territory of paper construction, then you can throw away a lot of this advice, and do what’s comfortable. However, you still need to keep your collaborators in the loop, and organize yourself long before the deadline for the paper. This is how I organize my thought process, so it is unlikely it will perfectly match what is best for anyone else. However, I’ve found that new students need guidance, and to understand a model for doing this.

Resources

I’ve provided copies in the library area the following books. Please make them available to others if you aren’t actively reading them.

• The Elements of Style, Fourth Edition by Strunk Jr., William. This is a classic book, and still provides good advice on how to maintain simplicity of writing, while effectively maintaining your point.
• Edit Yourself: A Manual for Everyone Who Works with Words by Bruce Ross-Larson A more detailed treatment of how to manage your own writing. How can you simplify sentences, manage their complexity, and be careful about their grammar.
• A Writer’s Guide to Transitional Words and Expressions by Pellegrino, Victor C. This books is especially useful if English is not your first language. How do you move your thoughts from one topic to another, while providing proper transitions? What are some of these transitions, and what is “good taste” around using them?

I highly suggest you buy a copy of each of these.

Some advice on systems paper construction:

• How (and How Not) to Write a Good Systems Paper
• How to write a systems paper
• How to get your systems papers accepted?
• Armando’s Paper Writing and Presentations Page
• Tips and Guidance for Students Writing Papers and Reports

Updates

• August 7, 2018 - Updated the Paper Organization Options section to 1. fix a typo, and 2. add more information about managing paper dependencies.

Nothing in the world can take the place of persistence. Talent will not; nothing is more common than unsuccessful men with talent. Genius will not; unrewarded genius is almost a proverb. Education alone will not; the world is full of educated derelicts. Persistence and determination alone are omnipotent.

• Calvin Coolidge

Learning how to do both research and system design and implementation is hard. We enter into a research project with some base-level of knowledge and capability¹. Most interesting problems – including earning a PhD – require you to increase your knowledge to the point that the problems are tractable. I’ve come a long way in my own understanding of what it takes to make myself productive, a process which include both gaining knowledge, and the capability to deploying it. This post covers my model for understanding the factors that go into this process.

Unfortunately, the means for systematically acquiring knowledge and capability are an area far from my expertise, so this post should be interpreted as opinion. This is a dump of my own mental models about the factors that go into being effective.

Knowledge Acquisition as Compounded Interest

I’ve watched enough students wrestle with understanding how systems work, and how to design them, to have confidence in the following assertion:

The amount of time that you put into research yields super-linear increases in capability with respect to the amount of time spent.

The more you know \(\to\)
the more context you have to understand concepts \(\to\)
the more you can make connections between concepts \(\to\)
the more you can understand \(\to\)
the better your model of the world is \(\to\)
the more you know.

In short, the more you know, the deeper and faster you can dive into a system, which accelerates your understanding and knowledge about the system. This creates a “virtuous cycle”² that must be leveraged to become a world expert in an area (i.e. completing a PhD).

A Model for Knowledge Gain

This knowledge gain (at some level of detail) seems quite similar to me to the impacts of compounded interest. The basic idea behind compounded interest is that a given annual percentage rate (APR) increase in your funds, will increase not only your principal investment, but also on any previous APR increases from previous years. The impacts of compounded interest are, somewhat unintuitively, very significant. The compounded interest formula, interpreted in the context of knowledge gain:

\(K = k' \times (1 + r/n)^{n \times t}\)

where:

• \(K\) = The “amount” of knowledge accumulated after \(t\) years.
• \(k'\) = The base level of knowledge at the start of the \(t\) years (analogous to the principal).
• \(r\) = The annual rate of knowledge gain (analogous to the APR).
• \(n\) = The number of “bursts” of learning throughout a year. I’ll simply and treat this as a monthly gain.
• \(t\) = The number of years over which you learn. I’ll focus on six years as the high-end of the historical average for a PhD.

I’ll get to where I think this model is wrong in a bit, but for now I’m going to discuss it’s utility. To make it concrete, I’ve plotted three different configurations:

• The blue line depicts someone with \(0.6\) knowledge gain per year (i.e. each year, they increase their knowledge by 60%, while starting at a base knowledge of \(1\)³.
• The red line depicts someone with a per-year knowledge acquisition of \(0.45\), or 75% the rate of the previous person.
• The green line represents someone who learns at the same rate as the red line (\(0.45\)), but starts off with double the amount of knowledge of the others (\(2\)).

The absolute value of “knowledge” for each of these is somewhat meaningless. Instead, I’ll focus on the relationships between the lines.

Impact of effort. The effort put in over time to better your knowledge and capability is represented by \(r\). The blue and red curves only differ in this parameter, and the red curve represents someone who puts in 75% of the effort of the blue. If someone works 6 hours a day for someone else’s 8, after 6 years, they will know less than half as much as the other. The intuition here is that relatively small, but consistent effort put in over time compounds to significant capability differences.

Impact of initial knowledge. Everyone comes into a domain with a base-level of knowledge. The green line represents someone with double the base knowledge of the other lines, but who puts in 75% of the effort of the blue line. The blue line passes and makes up for this initial disparity in knowledge and, given the slope of the line, strongly accelerates past the green. Note that sometimes having a higher base knowledge can structurally encourage lower knowledge-gain rates (see “Smart Kids” below).

Super-linear increases in knowledge. The exponent in the formula, if anything, is the key insight of the model. Note how quickly each curve increases in its rate of rise at around 3 years. Anyone who has done a PhD knows how quickly you feel like your progress increases at around this time.

Lies and Models

As I mentioned previously, I have no idea if this model is accurate or true. Sometimes we have to make models to understand ourselves and the world around us, and if they are outside of our domain of expertise, they could be completely wrong. Think of it as a hypothesis that needs to be tested and reflected on as we gain experience. That experience can refute the hypothesis, or it can allow us to gain some confidence in it.

In my experience, this model provides significant predictive capability. It predicted my experiences with myself and other PhD students when I was doing my doctorate. Most importantly, as I’ve mentored more and more students, it has predicted the outcomes of many – though not all – of them. More importantly, this model increases my own motivation to effectively climb the steep curve of knowledge acquisition.

If this model helps you, great. If not, shed it off, but make sure to ask yourself how you can effectively improve yourself in both the short term, and in the long term.

Time \(\neq\) Productivity

What factors go into any specific rate, \(r\), of knowledge acquisition? A clear factor is time. Spending sufficient amounts of time on self improvement will yield higher productivity, right? Well, no. We can use our time poorly, and knowledge gain does scale up perfectly with increased time expenditure. More importantly, if you try and optimize for \(r\) above all else, it will likely have the opposite effect.

If you try and fill your weeks with learning to optimize for \(r\), you’ll quickly find that \(r\) actually decreases. It is important to consider the relationship, \(r = f(h)\) where \(h\) is the hours per week you put into self-improvement. This is certainly not linear. In my experience it is logarithmic – you reach diminishing returns after a certain \(h\) – with a distinct “cliff”, \(c\) – where \(f(h > c) << f(h \leq c)\). See the section later on diminishing returns for an elaboration.

X Hours Toward Maximum Productivity

Just because you spend \(X\) hours working and researching, does not mean that you’re gaining an optimal amount of experience from those hours. I’ve previously discussed how to best use your time in the research domain. I’m only going to add one observation beyond that post.

“Smart” kids. In CS, there is a bad perception that students who will achieve the most, are those that start out with significantly higher knowledge than their peers. There’s a harmful stereotype about the geeks destined for success. Don’t buy into it. Of course, knowing more and being more capable is always preferable to the alternative, all other factors being the same. However, it is my experience that the rate of knowledge gain is higher for students who come in with significantly lower base knowledge. As the graph above shows, a higher rate can easily surmount base knowledge. I’ve observed two contributing factors to this:

Learning to fight for knowledge is hard. Research can be quite challenging, and should push anyone involved beyond their comfort zone. Some people have been able to leverage their innate intelligence and ability to get them to their current position. It can be difficult for these people to suddenly be quite challenged, and have to push their own knowledge acquisition rate up to compensate. In short, if you haven’t had a significant amount of practice having to fight for your knowledge, it can be difficult to learn how to summon that drive.

“Knowing” the answer is often counter-productive. Their high base knowledge is often accompanied by a high confidence. This confidence is, in many cases, quite justified given their past successful experiences. However, that confidence will not be justified in a domain that significantly pushes their capabilities. Research, by definition, should not be a domain approached with a sense that one either knows the answer, or can rely on their intuition to provide it. Difficult problems demand that you listen to your intuition, then you ignore it, and pivot to rigorously evaluate the situation. Intuitive, knee-jerk solutions often kill creativity, and rule out innovative solutions. Knowing “the answer” is counterproductive for finding the “best answer”.

A characteristic I’ve observed about people in this category: making assertions instead of asking questions. For example, a common behavior I’ve observed: someone doesn’t approach a technical discussion by asking questions, and instead provides assertions that (without some push-back) would kill further discussion. From a group perspective, I think this has a dangerous impact on effective discourse. I’ve seen this kill conversations about deep materials, stifle creativity, and result in ineffective communication that stunts the mentoring process. If someone approaches group discussions in this way, they likely approach their own inner monologue similarly.

I believe this can easily be remedied: prioritize asking questions, and foster a skepticism about intuitive assertions. A good way to do this, is to change your perspective on how to interact with others.

Diminishing and Negative Returns

The first six years of my higher education were characterized by what I think is an immature caricature of how to be productive. I valued hard work above almost anything else. I’ve always been work/life ratio-challenged as I generally enjoy the domain of my work so much, but this phase in my life took that to an extreme.

It took me two years of doctoral work to completely burn out, and realize that putting hours in after a certain point not only gave diminishing returns in terms of knowledge acquisition, but had negative returns. Spending an extra two hours in the lab on a Saturday inhibits the recovery of necessary “down time”. As it turns out, I can will myself to put outrageous amounts of effort in, but I cannot will myself to spend all of that time productively.

Constantly re-evaluating if you are spending enough time on the aspects of life that enable you to refresh and be productive is both important, and very personal. It took me a long time to converge on a pattern that allows me to be productive. Though I’ve aggressively pursued a knowledge gain curve with the highest possible slope, it was essential to recognize that the number of hours put in was not the only relevant variable. As part of understanding yourself, it is important for everyone to find all the variables that go into their own knowledge curve.

Summary

The take-aways:

1. Knowledge and capability increase super-linearly. It is hard for humans to understand exponential relationships, and aggressively super-linear relationships, in general. This means that you might look at a challenge that is impossible now, but in a couple of years it might be trivial.
2. Where you start out in your knowledge is less important than carefully ensuring that you grow over time. As intellectual growth is a compounding factor.
3. It is important to understand how to maintain a healthy rate of weekly time expenditure. Too little, and you aren’t taking advantage of the compounding factor, and tasks that seem hard will remain hard for a long time. Too much, and you reach diminishing returns due to cognitive limitations and a lack of work-life balance negatively impacting your growth.

So, it is important to contemplate what we each should do to effectively gain capability in our chosen fields. This is meant to increase introspection into how you might plan and manage your own self improvement, and gives you a rough model through which you can reflect on and plan your own development.

Quite a bit of information exists on the internet for how you should perform Pull-Requests (PRs), and how to conduct a code review. This post discusses one vision about how these software engineering practices fit into a research group that is building a significant body of code. I’m not an expert in any of these areas, but I have seen what works, and what doesn’t work in support of sustained research.

Why Software Engineering Exists

It is rarely obvious why software engineering practices are necessary or beneficial to developers that haven’t yet worked in large teams. These practices provide “processes” that restrict the methods and means of software development in a team. Let me say clearly: they are necessary, and they are beneficial. When someone is maturing as a software developer, they often realize that simply writing code isn’t the hard part. The hard part is writing code that is understandable, debuggable, and maintainable in the longer term. The goal is often for this code to be part of our shared infrastructure. Importantly, it is important to realize that the code you write will not be used only by yourself. Thus, you’re writing code that needs to be understandable, debuggable, and maintainable by others. That is a pretty high bar.

A big part of software engineering can be seen as a set of conventions and processes overlayed onto our normal process of coding. They are designed to produce code, documentation, and development histories that are high quality, and can be productively used by every member of the development team. At least that’s the goal.

In academia, we have a set of constraints that work against code quality (races for paper deadlines, and lots of perceived throw-away evaluation code), documentation (it doesn’t feed into someone’s PhD), and development histories (no mandated version control usage patterns, and little training in how to use version control). So I’m going to discuss a process that we apply to our group’s development, and how it attempts to enable a long-term, sustainable infrastructure. We don’t do this perfectly, but this post is meant to give the rational and expectations behind this process.

Many development teams in industry have extensive processes for development aside from the actual code. These include a fairly strict regime including commit, PR, and code-review guidelines, continuous integration and unit testing, automated bug tracking and lifetime management, scrum and structured meetings to structure short and long term goals of the group and individuals, etc… Though some of these have been adopted by academia (for example, scrum for academia), we are not necessarily creating a production-stable API as research requires agility, and we are a small team with members that are often working on separate projects, so don’t necessarily need as much cross-team coordination. However, the notions of curating our code through structured PRs, code reviews, and unit tests/continuous integration are necessary for the code-base to have any longevity. This document discusses the first two of these.

Team Cohesion, Code Quality, and Code Reviews

For a software infrastructure to have any longevity beyond a single student, a group needs to be involved in the shared development of the system. Having implemented a feature that improves the infrastructure, it is important to get some buy-in and quality control from the group before the code is pushed to mainline. The main goal of producing infrastructural code is to generate maintainable and readable code. Thus, it is naturally quite useful to get others from the group involved in the process of generating your code. This is a way to get a sanity check that the code that you wrote to be readable, actually is readable.

The main goals of our code reviews are to:

• build cohesion in the group as we see what each other are working on and learn about different parts of the code-base;
• improve the quality of our code by getting more people with different perspectives to examine it before it is in mainline; and
• ensure that the code is readable and properly follows conventions.

Given this, how does one actually conduct a code review? I typically think of it as requiring two passes.

Code review first pass. When doing a code review, what should you focus on? What are you looking for, and what should you spend your effort on when doing a review? The first pass should focus on the following.

• Is error checking always performed? If not, you must request it be added. No excuse for laziness; always check your error values eagerly.
• Are all of the relevant cases properly tested? Are tests completely lacking? Can they be more complete? How does the author have confidence that the code works?
• Are assumptions stated clearly, with appropriate assertions added to check them? Try and to understand these, and ask the author questions about them.
• Are the Composite style guidelines always followed as spelled out in the Composite Style Guide. If not, you must request changes. If there are confusions about style, first consult the guide, and if they persist then talk to me, and we can update the CSG.
• Is the code confusing? If conditions or expressions are convoluted, say so. Unduly confusing code should be clarified, simplified, or restructured. This is an important part of code craftsmanship, but it is sometimes difficult for the author to properly evaluate. The code review is an opportunity for another person to bring their context to bear. If the code is necessarily confusing, then a comment is required to explain it.
• Do the chosen names convey information about the code? This is another area where a different perspective is useful. A reviewer should insist that names convey useful information, usually by proposing better names. This is an area where being constructive and providing suggestions is much more helpful than just saying “choose a better name”. The author chose that name initially because presumably (if they are following the code craftsmanship and style guidelines) they thought it was good. Saying “it is bad” isn’t helpful given this; additional input is required.

Code review second pass. It is often surprising for people to know that the core of a review actually revolves around the first pass. Any additional effort often focuses on the structure of the code. Should files be broken up? Should functions be broken up? Should an abstraction be refactored? This list is obviously much smaller than the first, but it requires that you go through all of the code before you can make a reasonable mental model about both the code’s structure, and what other potential structures might be. I’m going to avoid going into what might constitute a better structure here, and leave that for the CSG and for a future post.

What Doesn’t a Code Review Provide?

Put simply, reviews provide neither testing, nor thorough evaluation of logic. It is expected that every time someone does a PR, it is for well-tested code that follows the guidelines for code craftsmanship and the conventions in the CSG. Solely reading code will not yield a deep enough understanding of the code to fess out complex logic problems, or ensure thorough testing. In this way, a code review is a process that focuses on issues (style, error checking, test coverage, and complexity) that are a proxy for refined, well-tested code. In this way, they are complementary to proper testing.

How to Think About Commits and Pull Requests

Before a code review is performed, a PR must be generated. It is very important to understand the responsibilities of a researcher submitting a PR, and the goals of the PR. To understand many of the goals and conventions around PRs, you have to understand how to use version-control. See the Appendix for tips on using git.

PR intentions and responsibilities. Using git “correctly” is one thing, but submitting code to be included in the main repository is another. You should submit a PR on your work when you reach a point of tested completion. You should plan any high-level feature that you’re adding as a set of successive, more manage-ably sized PRs. It does take some thought to figure out how to break your task into smaller constituent tasks, but this is something you really have to be thinking about anyway. It isn’t feasible to do a reasonable code review on a large PR because it requires too much accumulated mental state. This is similar to why we can’t just write thousands of lines of code at once; we must consume smaller, sequential problems. So if we take the code reviews into account, it becomes clear that we aren’t just writing code; we’re writing code in a digestible manner.

Studies have been performed that found that defects were most successfully found if code reviews are performed on 400 lines of code or fewer. It is not a good idea to send a huge PR and expect a good result. The likely result will be “please break this up into separate PRs that can be reviewed separately”.

It is very important to understand that when a researcher submits a PR, they are asking other people to spend their time on the author’s behalf (i.e. to help improve the code). Given this, any perceived laziness on behalf of the author is often not received well. Not spending the time to go through a researchers own diffs before submission to ensure that obvious errors are purged is disrespectful of the time of the people doing the review. It is inherently saying that the reviewer’s time is valued less than the researcher’s. This is, quite obviously, not OK. Don’t do this.

Our Group’s Pull Request Policies

Given this perspective on code reviews and PRs, our research group has a number of policies:

• Code craftsmanship must be adhered to if you’re doing a PR. The style guide (CSG) must be adhered to to even ask for a code review or issue a PR.
• You must fill out the check-boxed list for each PR that validates that you did, in fact, adhere to conventions and follow craftsmanship requirements. This is mainly a reminder, but if you check off a box, and haven’t done the indicated task, then you should not expect a friendly review.
• When you submit a PR, please make sure to @-mention your fellow researchers. A good general rule is to @-mention at least two colleagues.
• Every PR should be reviewed by at least one other colleague, and by the repo owner (me for Composite).
• Every researcher should do reviews. If you’re @-mentioned, and no-one else has done a review, strongly consider it. If someone is working in your area (in an area where you’re familiar or want to learn about the code), then ask to be @-mentioned.
• Please try and finish a code review within 48 hours of the PR. I’m particularly bad at this, but it is essential to give prompt feedback, and get good turn-around on code modifications.
• If the code modifications are urgent, please mention this by including the text URGENT in the first line of the PR. I can apply these PRs without review, but the responsibility is that such changes don’t break mainline, and that the author will integrate code review feedback in a subsequent, and proximate PR.
• Frequent PRs are much better than infrequent, larger PRs. 400 lines or less is the target. See the logic for this above. Past a certain size, there is no review that is possible, and the author simply has to go back and re-design their string of PRs by making them more fine-grained.

Appendix: Git Tips

What tools should you always consider when making code modifications and additions, and when integrating with other’s changes? With git, the obvious commands that you must integrate into your development cycle include:

• git diff, git diff --stat, and git status which all let you understand at different granularities what changes have been made. Before you commit code, you should likely execute all of these commands. You do this for two reasons. First, you want to make sure that only code modifications that you want in the commit, are in it. Each commit should be relatively focused, so it is important to catch this. Second, it is a good way to remind yourself what the commit contains so that you can better summarize it in the commit message. It is also useful to know that you can execute diff on ranges of commits to see what changed (see git log to see the list of commits).
• git commit -a, obviously. Do not, do not, use the m flag. You should write a “full form” message, not a single line. The focus of your message should be on 1. summarizing the commit in a single-line (which is displayed along with the commit in Github), and 2. an explanation of the details of the commit. The latter should explain the intention, and often includes an itemized list of the main aspects of the commit. @s should be used to reference any issues it addresses.
• git branch <b> and git co <b> to use and manage your branches. Get used to using branches. Branches are useful for when you’re trying to debug a problem, or implement a new feature. To see how they’re useful, it is important to realize that it is not uncommon to get distracted, and have to redirect your attention to another feature or bug. Without branches (and the ability to roll-back to before your work on the feature), you end up with a commit history of half-finished features intertwined with bug fixes and other features. This makes your PRs schizophrenic and difficult to review. So by default, you should always be developing on a non-mainline branch. When you go fix a bug, or work on a new feature, ask yourself if it deserves its own branch.
• git stash is an acknowledgment that we mess up with our branches. If you find that you’re somewhat accidentally developing a new feature, or fixing a new bug, but have a bunch of unstaged changes on the current branch, you can stash them, create a new branch, stash pop the changes into the new branch. I’m sure there are other ways to do this, but this example at the very least demonstrates how stash can be used to delay a current set of changes.
• git rebase enables you to update a branch to an updated master, and to “squash” multiple commits, and clean up your commit history. When the master progresses beyond the point where your branch split from it, you generally want to fast-forward merge your changes onto the new master. git rebase master is your friend here. It is generally superior to git merge as it maintains a cleaner (linear) history. git rebase -i allows you to rewrite history by combining different commits. This is very useful as it enables you to have a set of commits that tell a story, and don’t have a number of “in progress” commits. My suggestion is that you label your commits that you’ll likely want to get rid of in the future with TODO as a header on the title line. More information about this can be found here and here.

Updates

• 6/12/17 - Thanks Robert Gifford for some suggested improvements and edits.

Recently, two of my students (a PhD student, and an undergraduate) stated as fact something that I find deeply incorrect and unsettling. This has been a little bit of a wake-up call to me that I’m dropping the ball on some fundamental mentoring. This post will correct the (understandably) mistaken idea that we do experiments in our papers to show how well our system does. This is not correct. This post will also address some followup questions regarding which comparison cases we should choose for our publications, and how to have higher confidence in your evaluation.

We’re Awesome!

When you’ve just spent a year or two of your time doing a ton of system implementation, creating a theoretical framework around that implementation, and are ready to publish the work, it is understandable that the first inclination is to think “how can we show how awesome our work is!?” Alternatively, when reading other’s papers, it is easy to become disillusioned about the experimental results sections because one interpretation of them is that the authors are, of course, going to show how awesome their contributions are. A natural conclusion I’ve seen a few people make is that everyone is just showing how good their system is, so why should we do anything other than skim these sections?

I will make three arguments for why you should fight against the urge to view experimental results sections as opportunities for self-aggrandizement.

• Reviewer Interpretation. Each paper that we submit to a conference typically has between four and eight reviewers chosen from among the research community of peers whose job is to vet the submission. These are not the type of people that you want to try and sell snake oil to. They are looking at what your contributions are, if they are backed up by evidence, and if those contributions are significant enough for inclusion in the proceedings. In short, they care if your work is valuable, and scientifically sound. Work that doesn’t show all the trade-offs involved in your system will never meet the bar of “scientifically sound”. Evaluations that show only the positive aspects of the design, are not scientifically sound. They often make mistakes. But you should never submit a paper that relies on the mistakes of reviewers for acceptance.

• Reader Perception. When someone takes the time to sit down and read your paper, they are looking to trade their own (valuable) time, for knowledge. When they read an experimental section that paints a uniformly rosy picture of the system being introduced, they can and should be skeptical. Systems exist in complex environments that trade-off different resources and guarantees. Rarely, if ever, do we implement something that is uniformly an improvement over existing systems.

  An astute reader approaches a paper with a honed sense of skeptical analysis. If they’re presented with only a glowingly positive perspective on the contributions of a paper, they will likely feel that they did not attain a complete understanding of the system. They might feel that the time they spent with the system didn’t glean them the knowledge they hoped for as significant questions linger about the work. When I hear industry members talking about academic research, they often complain about the “unrealistic” experimental evaluations. We are not in the business of publishing only to publish. We want to convince people that our techniques are valuable in a well-defined domain.

• Scientific progress. Somewhat counter-intuitively, perhaps, just showing why your system is good does not add much to the scientific progress of a community. This is complicated because papers are generally not accepted when their contributions do not show benefit. “Negative results” have a very poor history of being published in systems, even when they do create knowledge (i.e. when one would expect a positive outcome, and do not find it).

  Learning that some technique has a number of positive contributions is important, but you have to ask what knowledge the reader walks away with. The progress of our ideas relies on scientific progress, not salesmanship.

The Goal: Less Awesome, More Knowledge Creation

A set of perspectives:

Scientific progress. The goal of the scientific evaluation of our systems is not to show how awesome they are. The goal is to understand how they behave, and to validate the contributions in spite of any down-sides to the system. We particularly care about how they behave in situations that the world cares about (for example, with relevant applications) as this goes toward the value of the system. A subset of the evaluation will certainly show the situations where the research is superior, but that doesn’t mean you should focus your evaluation only on those situations.

Holistic evaluation. When designing and implementing a system, we have hypothesis about what the contributions of that system will be. That is, how will it add to human knowledge and capability? The evaluation is there to either validate those contributions, or force you to reevaluate your hypothesis. This points out that the fundamental contributions of the paper must also contain the same level of nuance as the experiments. They are there just as much to implicitly state where your system is not making a contribution as they are to state where it provides value. The contributions are generally stated in the positive (what value you add, not where you don’t), and the evaluations validate these claims. However, the evaluations also make the implicit, negative aspects (i.e. trade-offs) of your contributions more explicit.

Motivations matter. Only papers that provide value are accepted into proceedings. Regardless of the motivation for the experimental evaluation section, if we eventually just end up showing how great our system is, why does it matter what the goal is? To be part of the scientific community, the path you take in doing your own research matters immensely. Why?

1. The peer-review system relies on all members evaluating contributions based on their actual value, not salesmanship. If this system falters, then the gap between academic research, and corporate whitepapers shrinks and eventually the public will lose trust in our ability to make real scientific progress as our one-sided arguments fail to produce actual value.
2. Reputation is important, and gaining a bad reputation by doing work that has less value than is purported in the paper will taint other’s view of your own research. This will negatively impact peer-review, grant, and industry perception of your work.
3. If you choose to do research in the global community, you should buy into the values of that community. Long-term progress is only possible via thorough, repeatable, scientific evaluations of our work. If the goal is only to push your own work and to gain fame, there are many other venues for that.

Motivations matter. The integrity of our own work and how it is conducted is as important as the research itself.

Evaluation of Applications and Comparisons

An aspect of how we show the trade-offs of our research, is which applications we choose to use as microscopes for our systems, and which systems we compare against. Instead of going into too many details and special cases here, some vague guidelines:

• Relevant. We should always use applications that the world cares about. We should always compare against systems that the world cares about. This could mean that they are commonly used in a domain of interest, or that the system (even if not broadly used) is the best exemplar of a state-of-the-art technique. Relevant applications and comparison cases show the reader that we care about showing the value of the system in the current context of the world. This is why so many systems compare against Linux. It is an immensely relevant system used in many domains, thus any value provided over it has a decent chance of being significant.
• Sound. Applications we use, and systems we compare against must not be unreasonably ham-strung. We should always take pains to provide “apples-to-apples” comparisons where possible. If a comparison is bound to be unfair in some way (but the comparison is still relevant), then it is always best to err on the side of unfairly benefiting the competing system. If it is impossible to show value under these conditions, then significant effort must be put into increasing the fairness of the comparison.
• Complete. The evaluation regime must be formulated to delve into all salient trade-offs of our system. Often the comparison systems are best at showing this as they might do better for some values of a system or evaluation parameter, and worse in others. Evaluations that focus only on the benefits of the system often fail in their completeness.

Microbenchmarks vs. Applications

It is often not clear to fledgling researchers why we require both microbenchmarks and complete applications, and what value both types of evaluation provide.

• Microbenchmarks. Microbenchmarks provide small insights into the overheads of very specific parts of the system (often the system’s atoms). If a number of operations are provided in the system, these investigate their costs in the relevant dimensions (e.g. cycles/op). As argued in the post on system atoms, they provide an upper-bound on what is possible with the system, so this bound must be thoroughly evaluated. Because of this, microbenchmarks must be complete in the sense that they thoroughly detail the space of possibility for the system (i.e. what is the maximum achievable performance). If microbenchmarks include comparisons to comparable systems, they must be sound. However, by definition, microbenchmarks are not relevant on their own. They are analogous to understanding the strength of concrete (the microbenchmark); yet that strength alone doesn’t tell you the level of structural integrity of a bridge (the application).

• Applications. Applications, as you may guess, provide the relevance to the evaluation. They demonstrate that the system can be used to provide value within the context of an application that, presumably, the world cares about in some way. These evaluations must also be sound. However, completeness is somewhat nuanced. When evaluating \(N\) applications, you’re almost explicitly only looking at the subset of system behaviors that are expressed during the application’s execution. In this way, completeness is often not entirely satisfied. One often sees multiple applications and competing systems evaluated within the same paper as an attempt to bolster the completeness of the evaluation. Within the confines of the page-limit, this should be a goal. However, there should be an acknowledgment that completeness is not achieved, and an qualitative argument about the bounds of conclusions that can be drawn from the applications is necessary.

Scientific Evaluation

It is somewhat natural to treat the evaluation that we perform in our papers as the most boring part of the whole research process. We started the research in the first place because we hypothesized that our design had specific benefits, and this evaluation is to validate our implementation of that design. In reality, most of the design and implementation we do in systems is driven by an artistic mix of creativity and past experience, and the evaluation is where we must follow a scientific process¹.

It is also important to understand when doing an evaluation that we (as humans) are somewhat flawed by confirmation bias. When we run an experiment and get a good result, we will tend to see it as bolstering our positive arguments for our system. However, experiments are not that simple; often a result that looks good can hide more nuance that requires deeper investigation. So what can we do to prevent results that look good from blinding us from the need for a deeper investigation?

Proper evaluation of our research requires the application of the scientific method.

As computer scientists (especially in systems), we create the world we study, which makes the nature of our research quite different from traditional scientific domains that aim to study physical phenomenon. This means that when building our systems, we aren’t necessarily pervasively applying the scientific method. When we switch over into evaluation, it is important that we switch over into “scientific mode” from “engineering mode”. The subset of the scientific method I want to focus on is the feedback loop between designing evaluations, forming hypothesis, conducting the evaluation, and interpreting the results (which starts the whole loop again).

• Design the experiment. What aspect of the system are you evaluating, and what is the best test to perform that evaluation? Often you’re attempting to understand the behavior of the system when varying some set of variables. What are those variables? Given that, you have x- and y-axis, and a number of lines in each graph that you can use to investigate up to three of those variables. Often there are more variables than that, so how will you break the evaluation into separate graphs. One of the graphs should attempt to summarize the overall contributions, while the rest should study the trade-offs of the system. For each of the graphs, we continue on to the next phase.

• Forming a hypothesis. This sounds simple, but this is the step that is the most important, and the easiest to overlook. You must ask yourself what you believe the results of the experiment will be, and why. This forces you to deeply contemplate the relevant system effects, and create a mental model of how they will mutually interact. If your graph is a typical x/y/number of lines plot, what should the shape of the lines be? Where are the inflection points? How will the lines react to changing variables relative to each other?

• Run the experiments. You must be very careful at this stage to ensure that you’re executing the system in a way that is consistent with the intentions of the design. You must only modify the variables that you’re supposed to vary according to the experiment’s design.

• Interpret the results. Do the results match your hypothesis? If they do not match (in any way), then you must determine why. There are two broad reasons: 1. your hypothesis does not reflect the actual effects and mechanisms of the system, or 2. your experiment shows a bug somewhere (in the design, in the implementation, or in the results). For the former, you need to understand why your mental model of the system is wrong, and go back to square one of designing the experiment. Redesign the experiment given your new understanding, reform hypothesis, and iterate. For the latter, it is debugging time.

This process takes a lot of time, especially if you’re hypothesis are off and you don’t understand the system as well as you need to. Each iteration takes quite a bit of effort and time, but is requires to ensure that your results are measuring and depicting what they should. Because of this, it is important to realize that:

The enemy of proper evaluation is a lack of time.

When I was a PhD student, I aimed to get all of my evaluation done at least two weeks in advance of a deadline, and that tended to leave enough time to iterate if my hypothesis was off, or if there were bugs to fix. However, this time should be larger if you have less confidence in the implementation, or if your dealing with a system you don’t know well.

TL;DR²

Evaluation of our systems deserves care and attention. Doing it poorly is the fastest way to significantly tarnish your reputation, and have very little impact on the world. It is necessary, as researchers in the global scientific community, to take our own, and our community’s scientific integrity seriously. As a graduate student, it is important to be systematic in your evaluation, follow the scientific method, and leave yourself the time so that both of these are possible.

Why are people interested in systems? Fundamentally, it is usually because they appreciate and value understanding how the technology around us works. If you don’t know much about systems, but they are vaguely interesting to you, it is hard to figure out how to “dive in”. As with anything that requires some technical mastery, there is a lot of learning to be done, but you want to find a reasonable entry point.

This post is essentially a list of entry points. You’ll likely stumble around a few of things, understand a thing or two, and not really understand most of it. At first. You’ll find that, over time, you’ll understand more and more.

The Systems Hacking Club is there to bridge the gap. When you find something that seems interesting, and you want to learn more, just ask! Use #slack, use the meeting, and talk to other club members. You’ll be amazed how quickly you will accelerate if you put in the time and are aggressive about learning.

Some news sites:

• Arstechnica - A great tech news site that isn’t too geeky, but is certainly deeper than generic news sources.
• Proggit - Decent tech-y news, but with that funky reddit smell. Comments often crap.
• YCombinator - Tech-y news feed that is a little bit more technical than proggit. Comments less crap.
• Anandtech - Quite a bit more hardware-focused, but does go deep into current hardware pieces.
• The Next Platform - A focus on the current evolution of systems to larger scales, more data, and less energy. Pretty technical, but you can still get a decent high-level idea about what’s going on.
• Linux Weekly News - Linux-specific news. The “kernel” pages go into pretty deep details, but are really great for context once you can understand them (sooner than you’d think!).

Some great blogs:

• Dan Luu - Smart dude.
• Paul Khuong - Smart dude.
• The Morning Paper - Smart dude. High-level overview of academic papers.
• Embedded in Academia - Smart dude. Talks about a lot of compiler tech.

Have any other sites that you think should be added? Post them below, and I might add them to the list.

Communication with other humans is difficult. Many interactions in academia revolve around either attempting to convey information to others, convincing others of your ideas, learning from others, or critically examining other’s ideas. All of these situations involve information and experience asymmetries – some individuals know more about an area or idea than others. The goal of communication is to increase the total knowledge of all individuals involved. When trying to convince others of your ideas, the goal is also to raise their knowledge to a level where they understand and agree with the value of your approach. This can be quite difficult, and contemplating how to do it effectively is necessary for researchers.

Why is Communication Essential for Research?

First, lets detour and discuss why we care about this. Academia often requires groups of researchers, all attacking a set of shared problems of interest, all approaching the problem from different perspectives. Why is this? The total context, knowledge, and experience of a group of individuals is inherently larger than that of an individual. Though this seems like a sufficient justification for researching in groups, I’ve seen many skeptical students. This skepticism comes from the fact that doing research requires a deep-dive into a topic, and the perception can be that once you have reached a sufficient depth, group interactions lose value. This perspective is dangerously ignorant of two facts:

1. Any research area requires knowledge in the intersection of a large number of other sub-areas, creating a large venn diagram of research areas. Even if you’re an expert in a few of these areas, others will teach you in their areas that will enhance your understanding in your original area. Ideas are not independent, and exist in an eco-system. Having a group that reflects that eco-system is necessary if we are to do strong research.
2. Creativity is required for research, and pushing only into the depths of a small set of areas removes one of the most powerful creative tools we have as humans: the ability to change the context we use to look at an idea. If you approach an area from only a single perspective, the set of ideas you’ll generate will start to stagnate over time. New perspectives from other subjects enables your mind to refocus on the problems you’re solving with new context which is a fundamental driver of creativity. Since research in systems is often the product of applying creativity to engineering, and studying it scientifically, you cannot afford to hinder your own creativity.

I hope that this debunks the idea of the effectiveness of the “lone wolf” researcher. At the very least, it should motivate a genuine commitment to being an active member in the broader research group. To foster a group environment where each member can benefit from these factors, see the post on psychological safety.

How to Effectively Communicate

It is deceptively simple to think that because we talk to other humans all day long, and have been doing so for most of our lives, we can effectively communicate in an academic environment. This is generally incorrect. I think it is important to first acknowledge that effective communication is really difficult, especially in deeply technical topics with large information asymmetries. Second, remember that these discussions should be relatively ego-less, thus the goal is not demonstrating knowledge or intelligence, rather growing the level of collective knowledge. In my experience, if someone perceives the goal of interactions as an opportunity to demonstrate knowledge, it ends up being a large waste of everyone’s time.

First, lets focus on some individual factors that we can each to be increase our effectiveness of communicating.

• Understand explicitly What is your goal for the interaction? If you have a meeting scheduled with someone, what is the point of it? What do you want the other member of the meeting to learn, or what do you want to learn from them? Without a set of goals, it is unlikely that a technical discussion will yield much useful information transfer. It is important for your goals to match those of the other parties; always make sure that meetings have a stated purpose.
• What do you need to prepare given your goals? What do you need to learn or organize to aid in achieving those goals? Especially if you’re trying to learn about a topic from someone else, doing some initial investigations will enhance your ability to learn, and utilize their time more effectively.

This list of things we can do independent of others to aid in communication is a very small. There isn’t much that you can do to optimize your communication with someone else while only taking yourself and your ideas into account. To most effectively communicate, you need to focus on the other party’s state. To emphasize this fact, a set of examples that show that the requirements on communication vary based on the other party much more than on your or the topic.

Explaining your research. You want to explain your research to

• Your mom.
• Your significant other.
• A high-school student taking their first programming class.
• A CS undergraduate freshman.
• A CS undergraduate senior.
• A CS PhD student in a different field.
• A CS PhD student in a similar field (i.e. in the lab).
• A venture capitalist.
• A CEO.
• A Dean of engineering (not just CS).
• A CS professor in a different area.
• An academic you meet at a conference.
• Your advisor.

Go through the list, and give a three to six sentence summary of the work you’re doing for each audience. If there is significant overlap in your explanation for many of these categories, you’re doing it wrong.

Understand how a subsystem works. You want to understand how a subsystem of a system that you’re unfamiliar with works, and you ask

• A highly technical engineer.
• A project manager.
• A PhD student.
• A professor that is removed from the details.

How do you state your questions? How do you react and followup to their answers? Again, this should likely deviate for each of these groups.

Explaining a research paper. You explain a paper to

• Your mom.
• Your research group.
• Your professor.
• A group of undergraduates.
• A group of your peers at a research conference.

Again, the methods you use should have significant deviation. There are many other examples, but I hope this set makes the point.

These examples demonstrate the pervasive wisdom that you should “know your audience”. This is likely one of the most important aspects of communicating effectively with other people. It takes genuine effort and thought to contemplate what your audience knows, and how to convey your information, or how to state your questions in a manner that is most effective.

Manufacturing Empathy¹

Many people associate empathy with a vague notion of sharing another’s feelings. I like a definition that is little bit more concrete:

Empathy is the capacity to understand or feel what another person is experiencing from within the other being’s frame of reference, i.e., the capacity to place oneself in another’s position.

Manufacturing empathy expands on the trite concept of knowing your audience. It is not a passive activity that occurs before you make a presentation, and instead an active activity that you perform while engaged in a community, or a conversation. The point is to always work toward a greater understanding about the other’s perspectives, knowledge, and understanding. This allows you to much more effectively state your concepts or questions in a manner that is understandable, and always advancing the conversation toward greater collective knowledge.

Manufacturing empathy is a tacit acknowledgment that it takes hard work, dedication, and concentration to effectively state your thoughts in a manner that will resonate with and be impactful on other humans. It is motivated by the fact that it is more important to focus on the other person’s understanding of the information, than on the information itself.

Building mental model of other’s state. One way to describe empathy, is that it is concerned with building descriptive mental models of the other’s state and mind. These mental models have a number of dimensions.

• Goals. What are the goals of the other person? Understanding where the goal of the conversation is to the other person is essential to being able to continuously move the conversation toward that goal. Without a goal, the conversation is aimless and is unlikely to meaningfully increase collective knowledge. It is also important to acknowledge early on if there is a goal disparity between the parties. If there is, that needs to be addressed before move on to other discussion.
• Interests. What are the interests of the other person? To fully engage someone and make them want to understand the points you’re making, you have to understand what they are interested in. Tailoring your explanations and questions toward those interests will maintain a higher level of engagement, thus increasing the chance that you’ll meet your goals.
• Knowledge. What does the other person understand about each of the pieces of required knowledge to convey a point? All information builds on the understanding from an existing base of knowledge. Explicitly building this base of knowledge is called “scaffolding”, and is required in any methodical delivery of complex ideas. Even when learning from another, helping build their scaffolding to understand what you understand is necessary.
• Expertise. What are the areas of expertise of the other person? A person’s areas of expertise are more specific than their general breadth of knowledge or interests, and if it is somewhat close to the area of conversation, it is often one of the best sources of analogies. Analogies are an effective way to explain some concepts, especially to those with very little knowledge in the areas you need them to understand. Analogies must be comfortably in the areas of knowledge of the other person if they are to be effective, and are most effective when they derive from their area of expertise.
• Confidence. What is your confidence in your mental model for each of the above dimensions? If you’re used to talking to someone, your models should be highly accurate. This is the benefit of mentoring models like the PhD student/advisor relationship – an hour meeting can be very productive as a majority of time is spent actually increasing collective knowledge. An initial meeting with an arbitrary high-school student is difficult as any mental models you have are not very reliable. This leads to the “interview” problem: it is very difficult to build a high-confidence, descriptive mental model of someone who is interviewing for a job in the meager time allocated to interacting with them.

So if manufacturing empathy requires an active effort to build and refine your mental models of the other people you’re interacting with, what actions should one take? Techniques for manufacturing empathy include:

• Ask more questions than you convey facts. It is much more important to understand what the other person is trying to convey and understand their state of mind, than to continuously convey knowledge. Most of a discussion should revolve around asking questions. This applies equally to the person that is attempting to convey information to the other, and the person that is naturally asking questions. Asking questions to understand where someone is coming from should always be the highest priority.

• Focus more on how people reply to your comments than on what you’re going to say next. How the other people react to your comments is your main source of input about their mental model. How deeply do they understand the area your having a conversation about? Do they understand the points that you’re trying to make? Many people make a point, hear a reply, and move on to the next point, without much of a reflection on how the reply reflects on the other person’s actual understanding. This is a waste of time for everyone involved. The most common response to someone’s reply to one of my questions, or to a point that was just made is often “what do you mean by that”.

• Be attentive to ambiguity. When the other person states something that is consistent with the understanding you want them to have, spend some effort making sure that there is no ambiguity in their reply. Especially in technical areas, it is not uncommon for a single assertion to have many different interpretations. When someone is building their knowledge-base, their questions and comments are often ill-stated because they don’t have the experience formulate a consistent reply. It is very easy to see the answer you want in an ambiguous statement. However, it is important to understand that ambiguity can hide a lack of understanding. Don’t move on to the next point, and instead discuss the question to build a better mental model.

• Repeat the points the other person is making in your own words. This is, in effect, a way to check the consistency between your model of the person’s understanding, and their actual understanding. When you restate in different terms what the other is trying to say, you’re expressing your model of what you think they know. You’re giving them a chance to debug that model, interactively.

• Slow down. Most people seem to think that flying from point to point is the way to have a productive discussion. It may be the case that you “cover” more material in a given amount of time by essentially enumerating facts. However, it has almost no value as it runs counter to increasing knowledge and understanding. Time has to be paid to making sure that both people’s understanding is consistent. Equally harmful to conveying knowledge and understanding is being distracted by tangents. Tangents have a tendency to make it hard to maintain a strong mental model of where the other person is at because they, by their very nature, only distract from the main goals of the conversation.

All of these mechanisms are methodically building feedback loops into the formation of your mental model of the other person. The other person provides more information to you about their own mental state, and you check this information’s consistency with your model, and refine the model as necessary, often through further interaction. This methodology is counter to a sense of confidence that we can convey complex information easily.

Provide Feedback to Enhance Other’s Empathy

It is the job of each member in a conversation to make sure that each individual achieves their goals. Put another way, your job goes beyond just to saying your piece, or learning what you want. It is important, once you approach your interactions from the perspective of building empathy, to understand that you must enable others to do the same. This is quite a bit easier than building empathy, and largely centers around providing constant feedback on what you believe the state of the conversation is. This takes a number of forms:

• When you don’t understand something, say so, and provide as much information about your confusion as possible. It is very common for researchers to feel like they “should” understand something, so they will act like they do. This is a waste of everyone’s time. It is important to state this feedback carefully and respectfully, as some people can take it as a personal attack that they aren’t explaining their idea well. If you’re talking to me, don’t pull your punches; be as direct as possible.
• When you do understand something, let the other person know, and restate or summarize their point to ensure that you’re both on the “same page”, or that your have a mutual understanding. This is exceedingly important because this is the signal that the conversation can move on to the next point. If you feel like some conversations are much longer than they need to be, and that explanations are drawn out, it is likely because it is ambiguous when you understand, and can move on. Professors are particularly bad at “over-explaining” concepts. This is because 1. we understand that discussion while failing to conveying ideas is a waste of time, and 2. we are used to explaining ideas to people of varying knowledge levels, so usually conservatively explain toward a broader range of knowledge levels.
• Clearly state when you want an elaboration. This differs from not understanding something, in that you have learned something that opens up a new area of interest. Sharing that you’d like to learn more about that is essential for the other person to build a mental model of your interests.
• Repeat your understanding of the other person’s points. Even if you believe that you’re in mutual understanding, it is often important to continuously provide feedback on the state of your own understanding of the other person’s points.

Practice

As with any skill, manufacturing empathy takes practice. Research groups often provide this opportunity in a few different ways:

• scrum and other forms of weekly meetings
• research meetings often centering around paper presentation
• more informal paper discussions
• research/poster days
• practice/mock paper presentations
• lunches with fellow students
• meetings with other students and with your advisor
• teaching and being a teaching assistant

It is important that these initiatives are student-motivated and run so that they have genuine momentum and longevity. Hopefully this provides yet another motivation to organize these events.

Beyond Academia

I’ve framed this discussion in the context of academia. The general technique goes far beyond this context. A number of examples, and I’ll let you expand from there:

• Job interviews. Your job is to always figure out not just what the answer to each question in a technical interview is, but also what the person sitting across from you wants out of the interview. Often answering “correctly” isn’t as important as satisfying the goals of that person.
• Teaching. Many developers continuously have to teach their colleagues, and be taught by them. Teaching is one of the rare opportunities where you not only can practice building empathy, but must.
• Getting things you want. We want things that only other people can enable us to have. Promotions, investments, funding, volunteers, etc… When you want something from someone, it is very important to build a strong mental model about what they want, what their incentives are, and what they prioritize.

Composite is a capability-based OS. There have been many capability-based systems in the past, spanning from those that used hardware-capabilities such as Hydra¹, to those that implement them in software such as KeyKOS, Eros, Fiasco, and seL4. An old, but still seminal overview of capabilities is Levy’s book. This post discusses capability-based systems in general, only with some references to Composite where a concrete example is required.

Operating Systems and Resource Access Abstractions

Operating systems must provide an interface that separates untrusted, user-level computation from raw hardware resources such as memory frames, the CPU processing power, interrupts, and device drivers. There are a great many ways to do this that expose trade-offs along multiple dimensions. Many OSes such as UNIX variants do this by providing high-level abstractions such as processes (with threads, signals, coordination schemes, IPC, and memory management), files (with the hierarchical file-system, mapping, and psuedo-files), and sockets (with networking stacks, protocols, and device options). A core of logic in the kernel intelligently spreads hardware resources among the processes to provide each abstraction. Where shared namespaces exist (i.e. enabling a way for two processes to address the same resource) such as the hierarchical file system namespace, processes have access to and can modify the same resources. That enables them to impact each other’s execution, for example by using the shared namespace for inter-process communication IPC. An example of this is printer spool files that are written by processes, and read by the printer daemon.

OS Security. Operations systems have historically primarily been concerned with providing simple and efficient access to resources. UNIX has been successful partially because it provided abstractions that could be relatively efficiently implemented in the kernel. The focus of UNIX was not on strictly controlling how data could flow through the system (i.e. confidentiality), or on the fine-grained control of isolation to produce fault tolerant and secure systems. The liberal use of (discretionary access control)[dac] file-system access control including the set uid bit have enabled privilege escalations that often compromise large amounts of a system’s data. The co-location of libraries and already bloated monolithic applications makes compromise expose significant fault propagation so that a fault/compromise can impact significant portions of a system. Put in more exacting terms, such systems do not focus on the Principle of Least Privilege (PoLP). This guiding principle for system design dictates that the full set of resources that a computation has access to should be no greater than that which is required for it to accomplish its goals. A little thought should make it obvious how much this does not apply to typical UNIX processes that have access to all of a user’s files and the network (called ambient authority, regardless if they are all necessary or not.

One approach to increasing the application of the PoLP in UNIX systems is to add enough bandages around existing systems until they provide more desirable guarantees. There has been a long history of this being done successfully. Android uses a user and FS subset to constrain the access of individual applications. Modern browsers separate into separate processes compositing and I/O processes from each of the tabs to increase the use of the PoLP. SeLinux attempts to add labels and information for to Linux to limit the flow of data through the system, thus providing multi-level security (MLS). The act of continually adding bandages is, never-the-less, increasing system complexity, and an up-hill battle.

OS adaptation and efficiency. A newer focus on capability-based systems is on their ability to provide controlled access to low-level resources (i.e. those that are less abstract than those in UNIX). This is useful because when you remove abstractions between resource requests and access, you’re cutting down on the code that is running, and removing all of the assumptions that code makes. This has the effect of enabling more efficient and more adaptive access to system resources. For example, in UNIX, mapping and unmapping memory into a process is done through a system of system calls that make a number of assumptions about what the abstraction must provide. Unmapping memory requires that the mappings for that memory be expelled from each (TLB) cache for each core in the system. This operation is immensely expensive, and generally unavoidable. To be clear, many of the guarantees that this provides are quite useful, and often what you want when executing as a process. However, it makes it impossible to implement scalable systems that can make different assumptions about how to manage memory mapping.

Why Use a Capability-based System?

Capability-based systems address the issues raised above of OS security. They are fundamentally implemented around the PoLP and around confinement – the ability to erect information flow barriers between different users/processes. Resource access and sharing must be tightly controlled according to specific, and restricting policies with a variety of desirable properties. This is one of the historical motivations of capability-based systems.

Capability-based systems address the issues around effectively harnessing system resources. They enable the safe and controlled access to resources that are very low-level. The expectation is that high-level abstractions such as processes, and UNIX-style mapping and unmapping are implemented in the system within the components via the low-level resource access. In this way, high-level abstractions are built on the those at a lower-level, which is often the foundation for good system design (see Hydra, above).

What is a Capability-based System?

The concept of a capability-based system is simple. Each user-level component accesses resources through a level of indirection. Resource accesses are made by providing a key, the possession of which denotes the ability to perform a set of operations on a resource. Keys can be passed (delegated) between components to enable the sharing of resources. For example, the ability to communicate with another component is only allowed through the possession of a key, and if a component passes that key to another, then they both can access the resource through it. Capabilities are these keys.

Capability tables implement the lookup \(ct_x(c_i) \to r_j\) where \(ct_x\) is component \(x\)’s capability table, \(c_i\) is the capability that component is attempting to access, and \(r_j\) is the resource associated with that capability. System calls to the kernel from a component can be viewed as \(syscall(ct_x, c_i, o)\) where \(o\) is a resource-specific operation (map, invoke, delete) to perform on the resource: \(o(ct_x(c_i))\). The kernel’s job is simply to implement (a) means for identifying the resource for a given capability, and (b) to implement the resource operations. Most kernels provide some version of (b), the operations to perform on their abstract resources, so the unique aspect of capability-based systems is (a).

Capability-based systems are interesting as they

1. constrain the set of resources that each component has access to, and
2. can be implemented efficiently as a means for controlling access to low-level (thus frequently accessed) resources.

If memory were of no concern, one could imagine each of the capability tables being implemented as a simple array, each entry holding a reference to the associated resource (or NULL), a tag denoting which type of resource is referenced, and a bitmap including which operations can be performed on the resource. It is clear that the overhead of looking up a resource in this system is nearly minimal (array indexing). It is less clear how useful how this is to control access to system resources and to provide confinement.

Capability Delegation and Revocation

So how are the capability tables of each component populated? This boils down to two factors:

1. How is a component given access to resources (i.e. how are resources added to a capability table)? We call this capability delegation.
2. How is a resource that a component has access to, removed from the component’s capability table (i.e. how are resources removed from a capability table)? We call this capability revocation.

Delegation. Abstractly, we can view delegation as providing at least \(D(ct_d, ct_s, c_d, c_s)\) which takes a capability \(c_s\) in the source capability table \(ct_s\) where \(ct_s(c_s) \to r\), and creates \(ct_d(c_d) \leftarrow r\) in the destination capability table (assuming that before the delegation \(ct_d(c_d) = NULL\)). Different systems will provide different operations that provide different arguments to this function, but the core is the same.

Observe a few key properties:

1. A component (with capability table \(ct_y\)) can only delegate access to resources that it has access to. Thus, there is no expansion of resource access through delegation.
2. The destination component will be able to access the resource (via \(ct_x(c_i)\)) as efficiently as the delegating component.
3. Delegation can be recursive, meaning the destination component can now be the source for a future delegation to another component.

Each capability often also has a permission set associated with it. This indicates the set of operations that can be performed on the resource through that component. Delegation often also takes an argument \(p\), which delegates those permissions to the destination component. If a capability’s permissions are \(perm(ct_s, c_s) \to p\), then \(D(ct_d, ct_s, c_d, c_s, p)\) is allowed if and only if \(p \subseteq perm(ct_s, c_s)\). Resource operations \(o(ct_s(c_s))\) are allowed only if \(o \in perm(ct_s, c_s)\). All of this has an important effect related to (2) above: a component can only delegate access rights to resources it has itself.

An interesting use of the permissions set is that delegation itself can be considered a permission set-controlled operation. For example, only if \(D \in perm(ct_s, c_s)\) can a component perform \(D(ct_d, ct_s, c_s, c_s, p)\). Thus a component can delegate access to a resource to another component, but prevent that component from delegating it itself. This enables specific components to be charged with managing the resources. A component wanting access to a resource might have to ask a specific component that mediates all access to the resource.

Revocation. The other side of the coin is how we can remove access to a resource from a component. This is often used for very mundane reasons. If a component is done executing (e.g. main returns, or exit is called), then we can use revocation to remove resource access, and reclaim resources (e.g. memory) that are only accessed by the terminating component.

The minimal implementation of revocation performs the following: \(R(ct_x, c_i)\) where before \(R\), \(ct_x(c_i) \to r\), and after, \(ct_x(c_i) \to NULL\). The difficult question is who should be allowed to revoke a capability from a component? This is actually a historically difficult question.

How a number of systems handle this:

• Some systems track which components have made a delegation to which other components (a per-resource tree of delegations), and revocation actually revokes all capabilities derived from a capability in the source component. Modern L4 variants and Barrelfish use this approach.
• Some systems use indirection of capability \(c_j\) access through another capability, \(c_i\). A delegating component maintains access to \(c_i\), and the main operation it can perform is \(R\) – to revoke \(c_j\). Eros is the main system that used this. Some versions implemented it with a “epoch” counter per capability and object. Revoking capabilities involves incrementing the objects epoch, and any resource access through a capability must have a matching epoch, or an error is thrown.
• The last option is that capability tables are themselves referenced as kernel resources through capability tables. The capability-based access to a capability table denotes some level of access to it, including added and removing capabilities to it. To maintain the confinement properties, all created capabilities must be a delegation from the component’s own capability tables. Put another way, a component can only create capabilities for resources they already have access to. Importantly, this enables revocation on any capability tables you have a capability to. Composite uses this model through what we call “higher-order resource tables”.

Implementing an OS as a Capability-based System

The post on the design of capability-based systems contains details on how these systems (and specifically Composite) are implemented. A key to understanding how a full operating system could be implemented using capability-based resource access, is understand what the resources are, and how they are composed together to create higher-level abstractions. For example, how are individual capabilities to memory pages composed to provide the expected semantics for fork? How are individual thread capabilities composed together with a component to provide system scheduling?

First, the kernel resources are all very low-level. They are close to actual hardware abstractions. This includes pages, virtual address spaces, threads, communication end-points, and interrupts. To build the higher-level abstractions we expect from fully-featured OSes, we simply create components with limited capability tables that give them access to exactly the resources they require (but no more – PoLP). When a component needs more resources, it typically ask for them (via a communication channel accessed through its capability table) from another component whose job is to manage and multiplex some set of resources.

The big question is one of bootstrapping. The initial component that gains control upon boot typically has access to all system resources. It must partition and delegate resources to the system resource managers (i.e. the main scheduler) so that they can continue to build the tower of abstraction.

Acknowledgments

Gregor Peach motivated this post by expressing confusion for the capability design article. Great feedback as I realized there was a huge scaffolding gap to understand the system.

Updates

• Cleaned up the links (thanks Luke Baier!).
• Fixed md \(\to\) html conversions (thanks Gregor Peach!).
• Cleaned up and clarified some areas (thanks Teo Georgiev!).
• Fixed a typo (thanks again Luke!).

Half a day, or half an hour?

One way to look at a job is what granularity of time slots you break your day into. Graham posits that most people in tech operate either on the manager’s schedule or the maker’s schedule. Managers segregate their work-day into hour/half-hour blocks based on meeting with others, sending significant emails, and coordinating with other people. Any conversation at this level should never take more than an hour as they are often decision-based, not based on information-building/dissemination. On the other hand, “makers” must focus on the creation of value. Engineering requires building a mental context that is deep enough to completely understand the problem at hand, or brainstorming a new design. Both activities are wasteful if done in hour stretches due to the massive context required. It is advisable to devote at least four hour, contiguous stretches to it.

Understanding the Engineer’s Schedule

Why does an engineer needs their time split into four-hour blocks? Understanding this is pretty fundamental. Faced with a large body of code, you have to understand it, build a mental model if its inter-relations, determine what the goal and means to the goal are, make the changes/additions, and debug those changes. This requires a deep understanding of the issues faced, the goals, and the software environment in which the changes are made in. Abandon all hope, ye programmers that go in hour spurts.

Given the need for large contiguous blocks of time, what threatens our ability to effectively utilize them? A great summary briefly investigates the impact of disruptions on programmer productivity. The most interesting reference explores how programmers recover from interrupted programming tasks¹. The big take-away is that after an interruption, a programmer takes 10-15 minutes to re-acquire their context, and start programming again. If a programmer gets an interruption four times an hour, they get nothing done. This is harrowing news, and implies that it is worth putting effort into understanding how to minimize these distractions.

To make the higher-order bits perfectly clear:

• Programmers really require significant contiguous time to be productive.
• An interruption during one of these contiguous blocks can cause up to a 15 minute loss of productivity.

The Researcher’s Schedule

Research is an activity that is a little more nuanced than “manager”-style, or “maker”-style schedules. On the one hand, we do significant system building. Writing papers also requires building significant context to craft a well thought-out product. This fits exactly into the “programmer” prototype. On the other hand, research is inherently a social activity. Meetings are required to

• discuss papers, and build the context of the entire research group,
• read papers, books, code, and other means of booting up and staying current in a research area,
• determine the focus of, and refine the story behind a specific research direction,
• discuss in-depth design trade-offs to determine a course of action, which requires everyone to build significant state, and
• build a relationship between the professor and each researcher with the intention of creating a productive long-term collaboration, and dissemination of knowledge about how to conduct research.

These activities motivate another granularity, the two hour chunk.

While taking classes, a researcher has yet another constraint on their time. Classes absorb a two to three hour chunk of time that requires deep attention. Additionally, if a researcher has Teaching Assistant (TA) duties, teaching classes consumes a comparable amount of time. Unique to being a TA, the researcher must learn to manage answering student emails in a timely manner without tanking productivity.

How is a researcher able to be productive in their own research when faced with this level of adversity? One of the first skills that every PhD student should focus on is being quite diligent in their own time management. A lack of productivity very quickly turns into a lack of research output.

Managing a Research Schedule

It is quite important to actively make your own schedule work for you. The responsibility is on the researcher to get the most out of every productive hour of their day. In the end, doing research is only justified by the product. The means to the end is integral, and doing a PhD is about understanding the means to learn how to do research. But finally, papers must get published, systems must get built, and thesis must get written. The output is necessary, but not sufficient for doing a PhD.

I think that it is good to have a number of goals when creating your schedule. First, shoot for three days with two four chunks. This may be unrealistic during the first years when you’re taking classes, but is still a good goal. On these days, aggressively avoid distractions, and focus on being as productive as possible. If you must schedule something on these days, try to place that meeting at the beginning, or the end of the day. Generally, attempt to bin distractions together either in the morning or evening. The foremost goal is to create the largest chunk of contiguous research time as possible. If you end up with small fragments of time due to time slots you have no control over (classes, TA duties, etc…), then plan the work for those slots to be an activity that best makes use of a slot of that size. This should all seem somewhat obvious, but it is important to make a conscious optimization pass over your schedule.

Hacks and Technology

Headphones. Use headphones in the lab. If one of your co-researchers has their headphones on, assume that they are trying to be productive, and that interrupting them will destroy their context. Thus, don’t do it. If you’re interested in coffee or lunch, use Slack or email to organize the outing. These types of activities are very important to build the group dynamics, but they should be integrated with a productive pattern for each researcher.

Reading papers as your “flex” slots. Reading a paper generally doesn’t require large slots of time. It is not difficult to put a paper down after reading it for 45 minutes, and come back to it in two days. Take notes in the margins where necessary to enable you to reboot into the paper’s context if necessary. Reading papers also fills in commute time, and time in-between other meetings quite well. If you need to understand the ins-and-outs of a paper, then this strategy might not be appropriate as it might require deeper attention, but realistically, most papers we read don’t require that deep of concentration.

Managing synchronous interactions. Synchronous interruptions are the productivity enemy of large-blocks of research time. Taking your brain away from the research in front of you to have a conversation with someone else, or to answer emails quickly is what tanks your ability to use a four hour stretch. It is important, therefore to insulate yourself from interruptions. There are two main strategies (that I know of) for this:

1. Cluster your necessary synchronous interactions as much as possible.
2. Use asynchronous interactions for collaborations where possible.

Clustering synchronous interactions. Meetings are necessary for a researcher’s long-term development. However, that doesn’t mean that 3 meetings throughout the week can’t be clustered together so that programming productivity for a single day suffers to preserve productivity for the rest of the days.

Even interactions with your advisor should abide by these rules. It is easy to think that you should want to meet with your advisor as quickly as possible, whenever they want to meet. Always remember that your advisor wants your research to be successful. In the long term, your research is the most successful when you are able to focus on creating systems. So if interruptions from your advisor are prohibiting your productivity, it is your responsibility to figure out a system that works.

It is worth having a discussion at the beginning of each semester to figure out what the lab-wide “meeting days” should be. These are the only days that meetings can be scheduled on (with exceptions permitted).

Synchronous \(\to\) asynchronous coordination. Technology helps here. Sometimes you are “blocked” on a problem and can’t make progress without solving it. If the only way to become unblocked is to talk to a fellow researcher, then you have few options other than to talk to them ASAP. However, many times interactions in a group of researchers are not quite as dire, and can be delayed. Technology to the rescue!

Using Slack. Slack enables questions to be asked and answered asynchronously, and enables them to be both answered and viewed by all of the interested researchers. It is an intermediate communication medium between physical meetings, and email. That is to say, that it is asynchronous, but enables some semblance of reactive conversations.

It is important to view the things you post on slack as not necessarily being viewed and replied to immediately. This means that you must provide enough context in your posts for someone who is not engrossed in your problem to relate, and provide productive replies. Please keep in mind that this is more context than you think it is at any point in time. If your posts are ambiguous, or are lacking enough specificity to enable a reasonable reply, then you’re wasting your own time in writing the post, and the time of everyone else in the channel that might read it and try and reply.

Slack messages that were not instigated by one of your own questions should be viewed as interruptions, thus should be avoided while programming. Don’t feel that you need to check slack whenever there is a new message. Check it when you aren’t in a sprint of productivity. Slack integrations for RSS, github, and Trello are useful notifications, but should not be indulged until you’re able to take the interrupt without losing significant context.

A note on public vs private messages in slack: Your default should always be to use public messages. Debugging bugs, learning about the infrastructure, and discussing design – these are all actions that should be public so that other researchers can chime in, and can learn from the discussion. If you have something that is personal and should not be public, then private messages are appropriate. You’ll find that little is actually private.

Using Trello. Trello adds value in three areas.

1. Collaborative project planning. When many researchers are involved in a research project, they must coordinate in their coding and writing toward the goal (often a submission). In the minimum, a single researcher is coordinating with their advisor.
2. Planning your research pushes. I’ve found that I cannot be productive in making code modification and additions if I don’t have a clear vision of what coding is involved. I’ve also found that I cannot have a good view of what is required to meet a deadline without understanding the set of tasks between where the code is now, and where it needs to be to execute the necessary experiments.
3. Context-sensitive conversations. Comments, cards, and checklists are tied to a specific tasks. This enables a more effective asynchronous interaction. The hard part of asynchronous communication is having a shared context and understanding of what is being discussed. Trello provides enough context that this asynchronous interaction is more directed and useful.

To best use Trello, it requires that you 1. include a significant number of tentative cards that represent the next month(ish) of work so that you can see the out-lay of work, 2. update cards (creating them, and updating those that exist) to reflect the work that is required, and that is being done., and 3. work to make sure that enough context is provided in each card so that anyone who is following the project will understand it.

Indications of Success

• Are you able to get 4 contiguous hours of research done a day?
• Are you able to get 2 days/week of 8 hours/day productive research?
• Are you able to cluster context-disrupting activities together most days?
• When you sit down to be productive, have you enabled yourself to do so? Do you know the concrete steps you’re going to take over the coming hours? Have you isolated the time allocation for research?
• Look over your past week, did you allocate the time as you intended? If not, why?
• Look over the past month, did you hit the productivity level you wanted/needed given your deadlines?

Bonus: Understanding your Advisor’s Schedule

It is important to understand the time allocations of the people who you have to collaborate with, and your professor is an important variable in this equation. Why does a professor interrupt you when they do, and when can you delay the distraction to protect one of your productive blocks of time? Systems professors that attempt to maintain their own ability to produce code have a mix of “management”-style task, and “programmer”-style tasks. They have some combination of:

• Proposals. To fund a lab of researchers requires a set of grants that are paying the researchers. Writing the documents that result in this funding (with around a 10% success-rate) takes time and deep concentration.
• Working with other researchers. Never forget that your advisor is the advisor for other students that are working on topics that are possibly significantly divergent form your own.
• Teaching. This is obvious.
• Preparing for classes. The lead-work that goes into conducting a successful class is non-trivial. Making the lectures takes time (around 4x the lecture time).
• Managing a class. Classes also need attention aside from the content. Orchestrating homeworks, grading of homeworks, labs, and lab content takes time and meetings.
• Advising students. Undergraduates and graduates require advising. This might be as a formal advisor, but often is also informal, based on the relationships that the professor has formed with specific students.
• Reviewing papers and proposals. Your advisor is likely part of program committees for conferences. They must review \(X\) papers, often with your help. This takes an enormous amount of time and is a mix of reading papers, and commenting on them in depth.
• Collaboration meetings. If your advisor collaborates with other labs or companies, chances are they have somewhat frequent meetings with them. These requires context, so they need at least an hour or two of preparation for an hour meeting.
• Programming. If your advisor values staying close to the research and problems, they are still programming. All of the problems above about four hours of unbroken concentration apply here.
• Committee work. CS departments must change based on student body, changes in the popularity of topics, identification of problems in the department/curriculum, and changes in the academic market. These changes are organized and orchestrated by committees within the department, which require meetings and email coordination between multiple faculty members.
• Bureaucratic bullshit. From payment of students, and hiring new students, to setting up new collaborations, or extending existing ones, your advisor likely has to interface with what I like to call “lawyerly-bullshit”. This is the deepest of wastes of time, yet also one of the most important things to spend time on.

When you receive an email from your advisor, realize that it is likely not temporally sensitive enough to require your to interrupt a programming session. If they stop by, or send you personal message on slack, then you’re unfortunately going to have to suffer an interrupt.

Given the variety of different activities your advisor is involved in, it should always be your assumption that they don’t remember the details of any specific programming situation that you’re in. This is not a valuation of the advisor on your work, rather a realistic fact given the set of commitments they have. Always provide context on trello, in slack, or in email for the issues you’re having. If you don’t it will waste not only your advisor’s time, but also your own time as the replies you will receive will likely not be about what you expect them to be about.